Move this repository to the Kinto github organisation and rename it kinto-core

[Natim]

The more I think about it the more it makes sense to me to do that transition at some point.

However we are not in a hurry. I am just adding a ticket there as we already started to think about it.

[glasserc]

I don't think I know Cliquet well enough to have a valid opinion, but cleaning up the documentation depends on this issue, so I'll offer my opinion.

I think that something like Cliquet could be a valuable abstraction/utility for Python services. Cliquet gives the Mozilla services team a "beachhead" from which to enforce common requirements/practices in Python-based services -- things like logging, monitoring (heartbeats? metrics?), exception handling (if we call another service and it fails, can we include information about the original failure in our failure message?), versioning. I have no idea if the services team is going to make other Python-based services, but I can't imagine that Kinto is going to be the only one.

I don't know if Cliquet, as it stands today, is the library/framework that I describe above. It may be that the current state of the project is too closely bound to Kinto's requirements.

To help figure out the relationship between Kinto and Cliquet: if Kinto needs something in Cliquet to change, does the Services team try to balance that against the needs of other hypothetical services? Or does it just make the change?

In my opinion, there are two good options:

1. Cliquet is really just part of Kinto. This might be true if Cliquet's been too infected by Kinto's requirements. In this case, I think we should just merge the Cliquet and Kinto repositories. There's no value in keeping them in separate repositories -- it just makes things more inconvenient.
2. Cliquet is a separate library. Even if Cliquet today isn't perfect, maybe it's possible to merge parts of it into Kinto and keep the rest as a standalone library, or even just a set of documentation describing the best practices that we think services should implement. In this case I think we should largely keep Cliquet as it is (as a separate repository), although of course there will be code that needs to move back and forth until we find the correct boundary.

My personal belief is that the second possibility is more likely, although it's hard to be sure since only one real project uses Cliquet so far, and it's impossible to be confident in a library without a lot of clients.

[Natim]

The second possibility is what we aim for.

We actually have multiple projects that already uses Cliquet:

• Readinglist
• Kinto
• Syncto

The idea is that you should use Kinto for your service until you have too many business rules that you need to process on server side and then you will use Cliquet to create this service.

[glasserc]

OK, in that case I don't think we should rename it kinto-core, and I think we should close this issue :)

[almet]

I'm re-opening the issue as I believe it would be fine to have readinglist and syncto depend from a kinto-core product.

It also solves other issues described in Kinto/kinto#560 by lowering the number of concepts a Kinto user has to deal with.

[almet]

Well, I cannot actually re-open the issue as I don't have the permissions anymore. Can anyone do this? (I believe we should discuss this matter in the next community meeting)

[leplatrem]

I'll add some thoughts here :)

Cliquet gives the Mozilla services team a "beachhead" from which to enforce common requirements/practices in Python-based services [...] I can't imagine that Kinto is going to be the only one.

A year and a half ago, we thought we were going to develop and deploy several storage services. The concept of a reusable launchpad was making a lot of sense.

Today, I'm not sure we'll use Cliquet in other contexts than Kinto (especially the REST resource part).
The projects @Natim mentions are not the best use-cases for Cliquet indeed. If the Readinglist project had to be resuscited we would do it with Kinto. Syncto leverages the Kinto ecosystem, so using a kinto-core would not be irrelevant.

I don't know if Cliquet, as it stands today, is the library/framework that I describe above. It may be that the current state of the project is too closely bound to Kinto's requirements.

Yes, it is. And it is not infected nor bound to Kinto requirements or features.

By the way, we had a use-case for Cliquet with the absearch service. But to avoid the numerous Pyramid dependencies, Tarek decided to implement it from scratch (it thus has different configuration format, no Heka logging output, etc.).

[...] lowering the number of concepts a Kinto user has to deal with.

We have a chance to build a community around Kinto, and probably not with Cliquet. We should make sure not to lower the potential of the former because of the latter ;)

I think this is the crucial point: Cliquet is not infected by Kinto, but Kinto (maintenance or documentation) is sometimes made complex because of Cliquet :)

[glasserc]

OK, now that I understand better what we're trying to do with kinto-core, I agree with this issue and would like to address it. Here are the list of things I think need to be changed:

• rename cliquet in source to kinto-core
• rename Github project from cliquet to kinto-core
• move kinto-core project from mozilla-services to Kinto organization
• update readthedocs name from cliquet.readthedocs.org to kinto-core.readthedocs.org (and also point it to the new github repo)
• update Travis and Coveralls URLs to point to kinto-core
• change references on servicedenuages.fr from talking about Cliquet to talking about Kinto-core -- actually maybe we should rewrite the cliquet articles entirely
• release version 1.0 of kinto-core (or should it be 4.0.0 to follow Cliquet's current 3.1.3?)
• update pypi to point at kinto-core
• update kinto to depend on kinto-core instead of cliquet

[almet]

Thanks for the listing!

change references on servicedenuages.fr from talking about Cliquet to talking about Kinto-core -- actually maybe we should rewrite the cliquet articles entirely

I think it's fine to let them as-is, maybe we should create a new article explaining the move and add notes to the old articles, rather than modifying the old articles.

release version 1.0 of kinto-core (or should it be 4.0.0 to follow Cliquet's current 3.1.3?)

I wasn't thinking about changing the major, but that might be a good idea to start fresh.

We should also make sure to let a repository at mozilla-services/cliquet explaining the move (maybe pointing the explanation article)

I believe we should let the old cliquet.readthedocs.org resource in place, for people who will have older versions of the documentation of kinto, which refers to it.

Otherwise, it looks good to me!

[Natim]

I think it should be kinto-core 4.0

[glasserc]

We should also rename mozilla-services/cliquet-fxa.

[glasserc]

Re: https://github.com/mozilla-services/cliquet/blob/master/cliquet/views/hello.py#L30. Does this mean that we are trying to remain compatible with settings files that refer to cliquet, or that we are trying to let people migrate from kinto-core to some other named project seamlessly? If it's the second, then I guess I need to rename it, but I'm not sure what it means.

[glasserc]

We're going to need a new logo too :)

[almet]

I believe this was done to remain compatible with settings files that refer to cliquet :)

[leplatrem]

actually maybe we should rewrite the cliquet articles entirely

I don't think so. We don't rewrite the past :) Maybe add a banner or a note in the top.

Does this mean that we are trying to remain compatible with settings files that refer to cliquet

We can break things at this level, no problem to get rid of cliquet here.

You can add to the tasks list :

• Remove reference to cliquet (imports) in every kinto plugins
• Bump major version of kinto (3.X)

We should also make sure to let a repository at mozilla-services/cliquet

I was also thinking of this. Leave the cliquet repo and add a big warning.
Edit: this would also allow us to do some maintenance on cliquet for a while until we upgrade the production servers to the new kinto version.

Also, shouldn't we move this whole repo to a kinto python module kinto.core instead (subfolder in kinto repo)? I was wondering how renaming the repo would solve the documentation problems we face. If we merge the two repos, then we can have only one source of docs. It is just an idea and needs to be challenged.

[glasserc]

Good catch about the plugins. I guess we have to bump the major version in Kinto because we expose the underlying cliquet version to plugins.

I like the package name kinto.core better than kinto_core, but wanted to keep the repos separate so that someone could only depend on the smaller one if they were building a service. (I didn't pursue trying to do any kind of import magic to make it live in a separate repo but under the package path kinto.core.) But now that you mention it, I wonder if that's valuable. What kinds of things would other services use? How much weight is added by the rest of the Kinto repository?

Looking at the kinto repository, I see:

• kinto/__main__.py: a main script
• kinto/__init__.py: a main WSGI app
• kinto/authorization.py: permissions inheritance code
• kinto/events.py: definition of the ServerFlushed event
• kinto/plugins/default_bucket: just one plugin
• kinto/config: code to write a default config to disk
• kinto/views: the actual Kinto code

Altogether, we're looking at something like 700 lines of code plus tests (on top of cliquet's 4600 plus tests). Most of that code is probably not relevant to most kinto-based sites, but there isn't so much of it, so maybe it's worth the gains in simplicity.

[tarekziade]

@leplatrem

By the way, we had a use-case for Cliquet with the absearch service.
But to avoid the numerous Pyramid dependencies, Tarek decided to implement it from scratch
(it thus has different configuration format, no Heka logging output, etc.).

I should mention that the main reason I did not use Cliquet or Kinto was because absearch is using one single JSON file as its database and the requirement was that we needed a simple flow for letting people editing that JSON file (not a database). With no kinto-admin back then, and 2 weeks to ship, building a simple python app on the top of the JSON file with a custom routing was what made sense.

Maybe I could have used Cliquet for Hekka logging but if I recall correctly Cliquet was pulling Postgres dependencies and many other stuff where I just needed a pure python app. I decided not to use it when it complained I did not have the postgres libs.

As of today I still think Cliquet is not suitable for absearch unless we add a/b testing features in it.

[almet]

For the plugins, I wonder if that would make sense to issue a new version of cliquet, which would be an empty shell, actually importing everything from kinto-core instead (I actually wonder if this is technically feasible when you have submodules / packages, but if it is, it might be a good way to keep the compat).

It kinda sounds like a "bad good idea", but I thought it was worth sharing :) Thoughts?

[glasserc]

I've opened glasserc/kinto#1 as a first draft of what this could look like.

I think that would be a clever backwards-compatibility hack, but the only plugins we'd need that for are cliquet-fxa and maybe cliquet-pusher, which I think will be pretty easy to update.

[Natim]

Ok so we are moving forward with the merge.
The rationale can be found in the thread: https://mail.mozilla.org/pipermail/kinto/2016-May/000119.html