Rationale

Cliquet is a toolkit to ease the implementation of HTTP microservices. It is mainly focused on data-driven REST APIs (aka CRUD).

Philosophy

  • KISS;
  • No magic;
  • Works with defaults;
  • Easy customization;
  • Straightforward component substitution.

Cliquet doesn’t try to be a framework: any project built with Cliquet will expose a well defined HTTP protocol for:

  • Collection and records manipulation;
  • HTTP status and headers handling;
  • API versioning and deprecation;
  • Errors formatting.

This protocol is an implementation of a series of good practices (followed at Mozilla Services and elsewhere).

The goal is to produce standardized APIs, which follow some well known patterns, encouraging genericity in clients code [1].

Of course, Cliquet can be extended and customized in many ways. It can also be used in any kind of project, for its tooling, utilities and helpers.

Features

It is built around the notion of resources: resources are defined by sub-classing, and Cliquet brings up the HTTP endpoints automatically.

Records and synchronization

  • Collection of records by user;
  • Optional validation from schema;
  • Sorting and filtering;
  • Pagination using continuation tokens;
  • Polling for collection changes;
  • Record race conditions handling using preconditions headers.

Generic endpoints

  • Hello view at root url;
  • Heartbeat for monitoring;
  • Batch operations;
  • API versioning and deprecation;
  • Errors formatting;
  • Backoff and Retry-After headers.

Toolkit

_images/cliquet-base.png

Cliquet brings a set of simple but essential features to build APIs.

  • Configuration through INI files or environment variables;
  • Pluggable storage and cache backends;
  • Pluggable authentication and user groups management;
  • Pluggable authorization and permissions management;
  • Structured logging;
  • Monitoring tools;
  • Profiling tools.

Pluggable components can be replaced by another one via configuration.

Dependencies

Cliquet is built on the shoulders of giants:

Everything else is meant to be pluggable and optional.

_images/cliquet-mozilla.png

Examples of configuration for a Cliquet application in production.

  • Basic Auth, FxA OAuth2 or any other source of authentication;
  • Default or custom class for authorization logics;
  • PostgreSQL for storage;
  • Redis for key-value cache with expiration;
  • StatsD metrics;
  • Sentry reporting via logging;
  • NewRelic database profiling (for development);
  • Werkzeug Python code profiling (for development).

A Cliquet application can change or force default values for any setting.

Built with Cliquet

Some applications in the wild built with Cliquet:

  • Reading List, a service to synchronize articles between devices;
  • Kinto, a service to store and synchronize schema-less data.
  • Please contact us to add yours.

Note

Applications built with Cliquet can store their data in several kinds of storage backends.

A Kinto instance can be used as a storage backend for a Cliquet application! See cloud storage.

_images/cliquet-kinto.png

Context

(to be done)

  • Cloud Services team at Mozilla
  • ReadingList project story
  • Firefox Sync
  • Cloud storage
  • Firefox OS User Data synchronization and backup

Vision

General

Any application built with Cliquet:

  • follows the same conventions regarding the HTTP API;
  • takes advantage of its component pluggability;
  • can be extended using custom code or Pyramid external packages;

Let’s build a sane ecosystem for microservices in Python!

Roadmap

The future features we plan to implement in Cliquet are currently driven by the use-cases we meet internally at Mozilla. Most notable are:

  • Notifications channel (e.g. run asynchronous tasks on events or listen for changes);
  • Attachments on records (e.g. Remote Storage compatibility);
  • Records generic indexing (e.g. streaming records to ElasticSearch).
  • ... come and discuss enhancements in the issue tracker!

Similar projects

  • Python Eve, built on Flask and MongoDB;
  • Please contact us to add more if any.

Since the protocol is language independant and follows good HTTP/REST principles, in the long term Cliquet should become only one among several server implementations.

Note

We encourage you to implement a clone of this project — using Node.js, Asyncio, Go, Twisted, Django or anything else — following the same protocol!

[1]Switch from custom protocol to JSON-API spec is being discussed.
[2]Currently, the clients code was not extracted from the client projects, such as RL Web client (React.js), Android RL sync (Java) or Firefox RL client (asm.js).
[3]See https://unhosted.org.