Filtering¶

Single value

• /collection?field=value

Minimum and maximum

Prefix field name with min_ or max_:

• /collection?min_field=4000

Multiple values

Prefix field with in_ and provide comma-separated values.

• /collection?in_status=1,2,3

Exclude

Prefix field name with not_:

• /collection?not_field=0

Exclude multiple values

Prefix field name with exclude_:

• /collection?exclude_field=0,1

Sorting¶

• /collection?_sort=-last_modified,field

Counting¶

In order to count the number of records, for a specific field value for example, without fetching the actual collection, a HEAD request can be used. The Total-Records response header will then provide the total number of records.

See batch endpoint to count several collections in one request.

Polling for changes¶

The _since parameter is provided as an alias for gt_last_modified.

• /collection?_since=1437035923844

When filtering on last_modified every deleted records will appear in the list with a deleted flag and a last_modified value that corresponds to the deletion event.

If the If-None-Match: "<timestamp>" request header is provided as described in the section about timestamps and if the collection was not changed, a 304 Not Modified response is returned.

Request:

GET /articles?_since=1437035923844 HTTP/1.1
Accept: application/json
Authorization: Basic bWF0Og==
Host: localhost:8000

Response:

HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: Backoff, Retry-After, Alert, Content-Length, ETag, Next-Page, Total-Records, Last-Modified
Content-Length: 436
Content-Type: application/json; charset=UTF-8
Date: Tue, 28 Apr 2015 12:08:11 GMT
Last-Modified: Mon, 12 Apr 2015 11:12:07 GMT
ETag: "1430222877724"
Total-Records: 2

{
    "data": [
        {
            "id": "dc86afa9-a839-4ce1-ae02-3d538b75496f",
            "last_modified": 1430222877724,
            "title": "MoCo",
            "url": "https://mozilla.com",
        },
        {
            "id": "23160c47-27a5-41f6-9164-21d46141804d",
            "last_modified": 1430140411480,
            "title": "MoFo",
            "url": "https://mozilla.org",
        },
        {
            "id": "11130c47-37a5-41f6-9112-32d46141804f",
            "deleted": true,
            "last_modified": 1430140411480
        }
    ]
}

Paginate¶

If the _limit parameter is provided, the number of records returned is limited.

If there are more records for this collection than the limit, the response will provide a Next-Page header with the URL for the Next-Page.

When there is no more Next-Page response header, there is nothing more to fetch.

Pagination works with sorting, filtering and polling.

Partial response¶

If the _fields parameter is provided, only the fields specified are returned. Fields are separated with a comma.

This is vital in mobile contexts where bandwidth usage must be optimized.

Nested objects fields are specified using dots (e.g. address.street).

Request:

GET /articles?_fields=title,url
Accept: application/json
Authorization: Basic bWF0Og==
Host: localhost:8000

Response:

HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: Backoff, Retry-After, Alert, Content-Length, ETag, Next-Page, Total-Records, Last-Modified
Content-Length: 436
Content-Type: application/json; charset=UTF-8
Date: Tue, 28 Apr 2015 12:08:11 GMT
Last-Modified: Mon, 12 Apr 2015 11:12:07 GMT
ETag: "1430222877724"
Total-Records: 2

{
    "data": [
        {
            "id": "dc86afa9-a839-4ce1-ae02-3d538b75496f",
            "last_modified": 1430222877724,
            "title": "MoCo",
            "url": "https://mozilla.com",
        },
        {
            "id": "23160c47-27a5-41f6-9164-21d46141804d",
            "last_modified": 1430140411480,
            "title": "MoFo",
            "url": "https://mozilla.org",
        }
    ]
}

List of available URL parameters¶

• <prefix?><field name>: filter by value(s)
• _since, _before: polling changes
• _sort: order list
• _limit: pagination max size
• _token: pagination token
• _fields: filter the fields of the records

Filtering, sorting, partial responses and paginating can all be combined together.

• /collection?_sort=-last_modified&_limit=100&_fields=title

HTTP Status Codes¶

• 200 OK: The request was processed
• 304 Not Modified: Collection did not change since value in If-None-Match header
• 400 Bad Request: The request querystring is invalid
• 401 Unauthorized: The request is missing authentication headers
• 403 Forbidden: The user is not allowed to perform the operation, or the resource is not accessible
• 406 Not Acceptable: The client doesn’t accept supported responses Content-Type
• 412 Precondition Failed: Collection changed since value in If-Match header

Validation¶

If the posted values are invalid (e.g. field value is not an integer) an error response is returned with status 400.

See details on error responses.

Conflicts¶

Since some fields can be defined as unique per collection, some conflicts may appear when creating records.

If a conflict occurs, an error response is returned with status 409. A details attribute in the response provides the offending record and field name. See dedicated section about errors.

Timestamp¶

When a record is created, the timestamp of the collection is incremented.

It is possible to force the timestamp if the specified record has a last_modified field.

If the specified timestamp is in the past, the collection timestamp does not take the value of the created record but is bumped into the future as usual.

HTTP Status Codes¶

• 200 OK: This record already exists, the one stored on the database is returned
• 201 Created: The record was created
• 400 Bad Request: The request body is invalid
• 401 Unauthorized: The request is missing authentication headers
• 403 Forbidden: The user is not allowed to perform the operation, or the resource is not accessible
• 406 Not Acceptable: The client doesn’t accept supported responses Content-Type
• 409 Conflict: Unicity constraint on fields is violated
• 412 Precondition Failed: Collection changed since value in If-Match header
• 415 Unsupported Media Type: The client request was not sent with a correct Content-Type

HTTP Status Codes¶

• 200 OK: The records were deleted
• 401 Unauthorized: The request is missing authentication headers
• 403 Forbidden: The user is not allowed to perform the operation, or the resource is not accessible
• 405 Method Not Allowed: This endpoint is not available
• 406 Not Acceptable: The client doesn’t accept supported responses Content-Type
• 412 Precondition Failed: Collection changed since value in If-Match header

HTTP Status Code¶

• 200 OK: The request was processed
• 304 Not Modified: Record did not change since value in If-None-Match header
• 401 Unauthorized: The request is missing authentication headers
• 403 Forbidden: The user is not allowed to perform the operation, or the resource is not accessible
• 406 Not Acceptable: The client doesn’t accept supported responses Content-Type
• 412 Precondition Failed: Record changed since value in If-Match header

Timestamp¶

When a record is deleted, the timestamp of the collection is incremented.

It is possible to force the timestamp by passing it in the querystring with ?last_modified=<value>.

If the specified timestamp is in the past, the collection timestamp does not take the value of the deleted record but is bumped into the future as usual.

HTTP Status Code¶

• 200 OK: The record was deleted
• 401 Unauthorized: The request is missing authentication headers
• 403 Forbidden: The user is not allowed to perform the operation, or the resource is not accessible
• 406 Not Acceptable: The client doesn’t accept supported responses Content-Type.
• 412 Precondition Failed: Record changed since value in If-Match header

Timestamp¶

When a record is created or replaced, the timestamp of the collection is incremented.

It is possible to force the timestamp if the specified record has a last_modified field.

For replace, if the specified timestamp is less or equal than the existing record, the value is simply ignored and the timestamp is bumped into the future as usual.

For creation, if the specified timestamp is in the past, the collection timestamp does not take the value of the created/updated record but is bumped into the future as usual.

HTTP Status Code¶

• 201 Created: The record was created
• 200 OK: The record was replaced
• 400 Bad Request: The record is invalid
• 401 Unauthorized: The request is missing authentication headers
• 403 Forbidden: The user is not allowed to perform the operation, or the resource is not accessible
• 406 Not Acceptable: The client doesn’t accept supported responses Content-Type.
• 409 Conflict: If replacing this record violates a field unicity constraint
• 412 Precondition Failed: Record was changed or deleted since value in If-Match header.
• 415 Unsupported Media Type: The client request was not sent with a correct Content-Type.

Read-only fields¶

If a read-only field is modified, a 400 Bad request error is returned.

Conflicts¶

If changing a record field violates a field unicity constraint, a 409 Conflict error response is returned (see error channel).

Timestamp¶

When a record is modified, the timestamp of the collection is incremented.

It is possible to force the timestamp if the specified record has a last_modified field.

If the specified timestamp is less or equal than the existing record, the value is simply ignored and the timestamp is bumped into the future as usual.

HTTP Status Code¶

• 200 OK: The record was modified
• 400 Bad Request: The request body is invalid, or a read-only field was modified
• 401 Unauthorized: The request is missing authentication headers
• 403 Forbidden: The user is not allowed to perform the operation, or the resource is not accessible
• 406 Not Acceptable: The client doesn’t accept supported responses Content-Type.
• 409 Conflict: If modifying this record violates a field unicity constraint
• 412 Precondition Failed: Record changed since value in If-Match header
• 415 Unsupported Media Type: The client request was not sent with a correct Content-Type.

HTTP Status Codes¶

• 200 OK: The request has been processed
• 400 Bad Request: The request body is invalid
• 50X: One of the sub-request has failed with a 50X status

About transactions¶

The whole batch of requests is executed under one transaction only.

In order words, if one of the sub-request fails with a 503 status for example, then every previous operation is rolled back.

Linux¶

sudo apt-get install python3.4-dev

OS X¶

brew install python3

Linux¶

On Debian / Ubuntu based systems:

apt-get install libffi-dev libssl-dev

On RHEL-derivatives:

yum install libffi-devel openssl-devel

OS X¶

Assuming brew is installed:

brew install libffi openssl pkg-config

Linux¶

On debian / ubuntu based systems:

apt-get install redis-server

or:

yum install redis

OS X¶

Assuming brew is installed, Redis installation becomes:

brew install redis

To restart it (Bug after configuration update):

brew services restart redis

Client libraries only¶

Install PostgreSQL client headers:

sudo apt-get install libpq-dev

Install Cliquet with related dependencies:

pip install cliquet[postgresql]

Full server¶

PostgreSQL version 9.4 (or higher) is required.

To install PostgreSQL on Ubuntu/Debian use:

sudo apt-get install postgresql-9.4

If your Ubuntu/Debian distribution doesn’t include version 9.4 of PostgreSQL look at the PostgreSQL Ubuntu and PostgreSQL Debian pages. The PostgreSQL project provides an Apt Repository that one can use to install recent PostgreSQL versions.

By default, the postgres user has no password and can hence only connect if ran by the postgres system user. The following command will assign it:

sudo -u postgres psql -c "ALTER USER postgres PASSWORD 'postgres';"

Cliquet requires UTC to be used as the database timezone, and UTF-8 as the database encoding. You can for example use the following commands to create a database named testdb with the appropriate timezone and encoding:

sudo -u postgres psql -c "ALTER ROLE postgres SET TIMEZONE TO 'UTC';"
sudo -u postgres psql -c "CREATE DATABASE testdb ENCODING 'UTF-8';"

Server using Docker¶

Install docker, for example on Ubuntu:

sudo apt-get install docker.io

Run the official PostgreSQL container locally:

postgres=$(sudo docker run -d -p 5432:5432 postgres)

(optional) Create the test database:

psql -h localhost -U postgres -W
#> CREATE DATABASE "testdb";

Tag and save the current state with:

sudo docker commit $postgres cliquet-empty

In the future, run the tagged version of the container

cliquet=$(sudo docker run -d -p 5432:5432 cliquet-empty)

...

sudo docker stop $cliquet

Setting the service in readonly mode¶

It is also possible to deploy a Cliquet service in readonly mode.

Instead of having settings to disable every resource endpoint, the readonly setting can be set:

cliquet.readonly = true

This will disable every resources endpoints that are not accessed with one of GET, OPTIONS, or HEAD methods. Requests will receive a 405 Method not allowed error response.

This setting will also activate readonly heartbeat checks for the permission and the storage backend.

Scheme, host and port¶

By default Cliquet does not enforce requests scheme, host and port. It relies on WSGI specification and the related stack configuration. Tuning this becomes necessary when the application runs behind proxies or load balancers.

Most implementations, like uwsgi, provide configuration variables to adjust it properly.

However if, for some reasons, this had to be enforced at the application level, the following settings can be set:

# cliquet.http_scheme = https
# cliquet.http_host = production.server:7777

Check the url value returned in the hello view.

Deprecation¶

Activate the service deprecation. If the date specified in eos is in the future, an alert will be sent to clients. If it’s in the past, the service will be declared as decomissionned.

# cliquet.eos = 2015-01-22
# cliquet.eos_message = "Client is too old"
# cliquet.eos_url = http://website/info-shutdown.html

Logging with Heka¶

Mozilla Services standard logging format can be enabled using:

cliquet.logging_renderer = cliquet.logs.MozillaHekaRenderer

With the following configuration, all logs are redirected to standard output (See 12factor app):

[loggers]
keys = root

[handlers]
keys = console

[formatters]
keys = heka

[logger_root]
level = INFO
handlers = console
formatter = heka

[handler_console]
class = StreamHandler
args = (sys.stdout,)
level = NOTSET

[formatter_heka]
format = %(message)s

Handling exceptions with Sentry¶

Requires the raven package, or Cliquet installed with pip install cliquet[monitoring].

Sentry logging can be enabled, as explained in official documentation.

Monitoring with StatsD¶

Requires the statsd package, or Cliquet installed with pip install cliquet[monitoring].

StatsD metrics can be enabled (disabled by default):

cliquet.statsd_url = udp://localhost:8125
# cliquet.statsd_prefix = cliquet.project_name

Monitoring with New Relic¶

Requires the newrelic package, or Cliquet installed with pip install cliquet[monitoring].

Enable middlewares as described here.

New-Relic can be enabled (disabled by default):

cliquet.newrelic_config = /location/of/newrelic.ini
cliquet.newrelic_env = prod

Filtering¶

It is possible to filter events by action and/or resource name. By default actions create, update and delete are notified for every resources.

cliquet.event_listeners.redis.actions = create
cliquet.event_listeners.redis.resources = article comment

Backend¶

cliquet.cache_backend = cliquet.cache.redis
cliquet.cache_url = redis://localhost:6379/0
cliquet.cache_prefix = stack1_

# Control number of pooled connections
# cliquet.storage_pool_size = 50

See cache backend documentation for more details.

Headers¶

It is possible to add cache control headers on a particular resource for anonymous requests. The client (or proxy) will use them to cache the resource responses for a certain amount of time.

By default, Cliquet indicates the clients to invalidate their cache (Cache-Control: no-cache).

cliquet.mushroom_cache_expires_seconds = 3600

Basically, this will add both Cache-Control: max-age=3600 and Expire: <server datetime + 1H> response headers to the GET responses.

If setting is set to 0, then the resource follows the default behaviour.

CORS¶

By default, CORS headers are cached by clients during 1H (Access-Control-Max-Age).

The duration can be set from settings. If set to empty or to 0, the header is not sent to clients.

cliquet.cors_max_age_seconds = 7200

Authentication setup¶

Cliquet relies on pyramid multiauth to initialize authentication.

Therefore, any authentication policy can be specified through configuration.

For example, using the following example, Basic Auth, Persona and IP Auth are enabled:

multiauth.policies = basicauth pyramid_persona ipauth

multiauth.policy.ipauth.use = pyramid_ipauth.IPAuthentictionPolicy
multiauth.policy.ipauth.ipaddrs = 192.168.0.*
multiauth.policy.ipauth.userid = LAN-user
multiauth.policy.ipauth.principals = trusted

Similarly, any authorization policies and group finder function can be specified through configuration in order to deeply customize permissions handling and authorizations.

Basic Auth¶

basicauth is mentioned among multiauth.policies by default.

multiauth.policies = basicauth

By default, it uses an internal Basic Auth policy bundled with Cliquet.

In order to replace it by another one:

multiauth.policies = basicauth
multiauth.policy.basicauth.use = myproject.authn.BasicAuthPolicy

Custom Authentication¶

Using the various Pyramid authentication packages, it is possible to plug any kind of authentication.

(Github/Twitter example to be done)

Firefox Accounts¶

Enabling Firefox Accounts consists in including cliquet_fxa in configuration, mentioning fxa among policies and providing appropriate values for OAuth2 client settings.

See mozilla-services/cliquet-fxa.

Backend¶

cliquet.permission_backend = cliquet.permission.redis
cliquet.permission_url = redis://localhost:6379/1

# Control number of pooled connections
# cliquet.permission_pool_size = 50

See permission backend documentation for more details.

Resources¶

ACEs are usually set on objects using the permission backend.

It is also possible to configure them from settings, and it will bypass the permission backend.

For example, for a resource named “bucket”, the following setting will enable authenticated people to create bucket records:

cliquet.bucket_create_principals = system.Authenticated

The format of these permission settings is <resource_name>_<permission>_principals = comma,separated,principals.

See shareable resource documentation for more details.

Plug custom model¶

In order to customize the interaction of a HTTP resource with its storage, a custom model can be plugged-in:

from cliquet import resource


class TrackedModel(resource.Model):
    def create_record(self, record, parent_id=None, unique_fields=None):
        record = super(TrackedModel, self).create_record(record,
                                                         parent_id,
                                                         unique_fields)
        trackid = index.track(record)
        record['trackid'] = trackid
        return record


class Payment(resource.UserResource):
    default_model = TrackedModel

Relationships¶

With the default model and storage backend, Cliquet does not support complex relations.

However, it is possible to plug a custom model class, that will take care of saving and retrieving records with relations.

In Pyramid views¶

In Pyramid views, a request object is available and allows to use the storage configured in the application:

from cliquet import resource

def view(request):
    registry = request.registry

    flowers = resource.Model(storage=registry.storage,
                             collection_id='app:flowers')

    flowers.create_record({'name': 'Jonquille', 'size': 30})
    flowers.create_record({'name': 'Amapola', 'size': 18})

    min_size = resource.Filter('size', 20, resource.COMPARISON.MIN)
    records, total = flowers.get_records(filters=[min_size])

    flowers.delete_record(records[0])

Outside views¶

Outside views, an application context has to be built from scratch.

As an example, let’s build a code that will copy a collection into another:

from cliquet import resource, DEFAULT_SETTINGS
from pyramid import Configurator


config = Configurator(settings=DEFAULT_SETTINGS)
config.add_settings({
    'cliquet.storage_backend': 'cliquet.storage.postgresql'
    'cliquet.storage_url': 'postgres://user:pass@db.server.lan:5432/dbname'
})
cliquet.initialize(config, '0.0.1')

local = resource.Model(storage=config.registry.storage,
                       parent_id='browsing',
                       collection_id='history')

remote = resource.Model(storage=config_remote.registry.storage,
                        parent_id='',
                        collection_id='history')

records, total = in remote.get_records():
for record in records:
    local.create_record(record)

Resource¶

class cliquet.resource.UserResource(request, context=None)¶

Base resource class providing every endpoint.

default_viewset¶

Default cliquet.viewset.ViewSet class to use when the resource is registered.

alias of ViewSet

default_model¶

Default cliquet.resource.model.Model class to use for interacting the cliquet.storage and cliquet.permission backends.

alias of Model

mapping = <cliquet.resource.schema.ResourceSchema object at 140115463728080 (named )>¶

Schema to validate records.

collection¶

The collection property.

get_parent_id(request)¶

Return the parent_id of the resource with regards to the current request.

Parameters:	request – The request used to create the resource.
Return type:	str

is_known_field(field)¶

Return True if field is defined in the resource mapping.

Parameters:	field (str) – Field name
Return type:	bool

collection_get()¶

Model GET endpoint: retrieve multiple records.

Raises:	HTTPNotModified if If-None-Match header is provided and collection not modified in the interim.
Raises:	HTTPPreconditionFailed if If-Match header is provided and collection modified in the iterim.
Raises:	HTTPBadRequest if filters or sorting are invalid.

collection_post()¶

Model POST endpoint: create a record.

If the new record conflicts against a unique field constraint, the posted record is ignored, and the existing record is returned, with a 200 status.

Raises:	HTTPPreconditionFailed if If-Match header is provided and collection modified in the iterim.

See also

Add custom behaviour by overriding cliquet.resource.UserResource.process_record()

collection_delete()¶

Model DELETE endpoint: delete multiple records.

Raises:	HTTPPreconditionFailed if If-Match header is provided and collection modified in the iterim.
Raises:	HTTPBadRequest if filters are invalid.

get()¶

Record GET endpoint: retrieve a record.

Raises:	HTTPNotFound if the record is not found.
Raises:	HTTPNotModified if If-None-Match header is provided and record not modified in the interim.
Raises:	HTTPPreconditionFailed if If-Match header is provided and record modified in the iterim.

put()¶

Record PUT endpoint: create or replace the provided record and return it.

Raises:	HTTPPreconditionFailed if If-Match header is provided and record modified in the iterim.

Note

If If-None-Match: * request header is provided, the PUT will succeed only if no record exists with this id.

See also

Add custom behaviour by overriding cliquet.resource.UserResource.process_record().

patch()¶

Record PATCH endpoint: modify a record and return its new version.

If a request header Response-Behavior is set to light, only the fields whose value was changed are returned. If set to diff, only the fields whose value became different than the one provided are returned.

Raises:	HTTPNotFound if the record is not found.
Raises:	HTTPPreconditionFailed if If-Match header is provided and record modified in the iterim.

See also

Add custom behaviour by overriding cliquet.resource.UserResource.apply_changes() or cliquet.resource.UserResource.process_record().

delete()¶

Record DELETE endpoint: delete a record and return it.

Raises:	HTTPNotFound if the record is not found.
Raises:	HTTPPreconditionFailed if If-Match header is provided and record modified in the iterim.

process_record(new, old=None)¶

Hook for processing records before they reach storage, to introduce specific logics on fields for example.

def process_record(self, new, old=None):
    new = super(MyResource, self).process_record(new, old)
    version = old['version'] if old else 0
    new['version'] = version + 1
    return new

Or add extra validation based on request:

from cliquet.errors import raise_invalid

def process_record(self, new, old=None):
    new = super(MyResource, self).process_record(new, old)
    if new['browser'] not in request.headers['User-Agent']:
        raise_invalid(self.request, name='browser', error='Wrong')
    return new

Parameters:	new (dict) – the validated record to be created or updated. old (dict) – the old record to be updated, None for creation endpoints.
Returns:	the processed record.
Return type:	dict

apply_changes(record, changes)¶

Merge changes into record fields.

Note

This is used in the context of PATCH only.

Override this to control field changes at record level, for example:

def apply_changes(self, record, changes):
    # Ignore value change if inferior
    if record['position'] > changes.get('position', -1):
        changes.pop('position', None)
    return super(MyResource, self).apply_changes(record, changes)

Raises:	HTTPBadRequest if result does not comply with resource schema.
Returns:	the new record with changes applied.
Return type:	dict

Schema¶

class cliquet.resource.schema.ResourceSchema(*arg, **kw)¶

Base resource schema, with Cliquet specific built-in options.

class Options¶

Resource schema options.

This is meant to be overriden for changing values:

class Product(ResourceSchema):
    reference = colander.SchemaNode(colander.String())

    class Options:
        unique_fields = ('reference',)

unique_fields = ()¶

Fields that must have unique values for the user collection. During records creation and modification, a conflict error will be raised if unicity is about to be violated.

readonly_fields = ()¶

Fields that cannot be updated. Values for fields will have to be provided either during record creation, through default values using missing attribute or implementing a custom logic in cliquet.resource.UserResource.process_record().

preserve_unknown = False¶

Define if unknown fields should be preserved or not.

For example, in order to define a schema-less resource, in other words a resource that will accept any form of record, the following schema definition is enough:

class SchemaLess(ResourceSchema):
    class Options:
        preserve_unknown = True

ResourceSchema.is_readonly(field)¶

Return True if specified field name is read-only.

Parameters:	field (str) – the field name in the schema
Returns:	True if the specified field is read-only, False otherwise.
Return type:	bool

class cliquet.resource.schema.PermissionsSchema(*args, **kwargs)¶

A permission mapping defines ACEs.

It has permission names as keys and principals as values.

{
    "write": ["fxa:af3e077eb9f5444a949ad65aa86e82ff"],
    "groups:create": ["fxa:70a9335eecfe440fa445ba752a750f3d"]
}

class cliquet.resource.schema.TimeStamp(*arg, **kw)¶

Basic integer schema field that can be set to current server timestamp in milliseconds if no value is provided.

class Book(ResourceSchema):
    added_on = TimeStamp()
    read_on = TimeStamp(auto_now=False, missing=-1)

schema_type¶

alias of Integer

title = 'Epoch timestamp'¶

Default field title.

auto_now = True¶

Set to current server timestamp (milliseconds) if not provided.

missing = None¶

Default field value if not provided in record.

class cliquet.resource.schema.URL(*arg, **kw)¶

String field representing a URL, with max length of 2048. This is basically a shortcut for string field with ~colander:colander.url.

class BookmarkSchema(ResourceSchema):
    url = URL()

schema_type¶

alias of String

Model¶

class cliquet.resource.Model(storage, id_generator=None, collection_id='', parent_id='', auth=None)¶

A collection stores and manipulate records in its attached storage.

It is not aware of HTTP environment nor protocol.

Records are isolated according to the provided name and parent_id.

Those notions have no particular semantic and can represent anything. For example, the collection name can be the type of objects stored, and parent_id can be the current user id or a group where the collection belongs. If left empty, the collection records are not isolated.

id_field = 'id'¶

Name of id field in records

modified_field = 'last_modified'¶

Name of last modified field in records

deleted_field = 'deleted'¶

Name of deleted field in deleted records

timestamp(parent_id=None)¶

Fetch the collection current timestamp.

Parameters:	parent_id (str) – optional filter for parent id
Return type:	integer

get_records(filters=None, sorting=None, pagination_rules=None, limit=None, include_deleted=False, parent_id=None)¶

Fetch the collection records.

Override to post-process records after feching them from storage.

Parameters:	filters (list of cliquet.storage.Filter) – Optionally filter the records by their attribute. Each filter in this list is a tuple of a field, a value and a comparison (see cliquet.utils.COMPARISON). All filters are combined using AND. sorting (list of cliquet.storage.Sort) – Optionnally sort the records by attribute. Each sort instruction in this list refers to a field and a direction (negative means descending). All sort instructions are cumulative. pagination_rules (list of list of cliquet.storage.Filter) – Optionnally paginate the list of records. This list of rules aims to reduce the set of records to the current page. A rule is a list of filters (see filters parameter), and all rules are combined using OR. limit (int) – Optionnally limit the number of records to be retrieved. include_deleted (bool) – Optionnally include the deleted records that match the filters. parent_id (str) – optional filter for parent id
Returns:	A tuple with the list of records in the current page, the total number of records in the result set.
Return type:	tuple

delete_records(filters=None, parent_id=None)¶

Delete multiple collection records.

Override to post-process records after their deletion from storage.

Parameters:	filters (list of cliquet.storage.Filter) – Optionally filter the records by their attribute. Each filter in this list is a tuple of a field, a value and a comparison (see cliquet.utils.COMPARISON). All filters are combined using AND. parent_id (str) – optional filter for parent id
Returns:	The list of deleted records from storage.

get_record(record_id, parent_id=None)¶

Fetch current view related record, and raise 404 if missing.

Parameters:	record_id (str) – record identifier parent_id (str) – optional filter for parent id
Returns:	the record from storage
Return type:	dict

create_record(record, parent_id=None, unique_fields=None)¶

Create a record in the collection.

Override to perform actions or post-process records after their creation in storage.

def create_record(self, record):
    record = super(MyModel, self).create_record(record)
    idx = index.store(record)
    record['index'] = idx
    return record

Parameters:	record (dict) – record to store parent_id (str) – optional filter for parent id unique_fields (tuple) – list of fields that should remain unique
Returns:	the newly created record.
Return type:	dict

update_record(record, parent_id=None, unique_fields=None)¶

Update a record in the collection.

Override to perform actions or post-process records after their modification in storage.

def update_record(self, record, parent_id=None,unique_fields=None):
    record = super(MyModel, self).update_record(record,
                                                parent_id,
                                                unique_fields)
    subject = 'Record {} was changed'.format(record[self.id_field])
    send_email(subject)
    return record

Parameters:	record (dict) – record to store parent_id (str) – optional filter for parent id unique_fields (tuple) – list of fields that should remain unique
Returns:	the updated record.
Return type:	dict

delete_record(record, parent_id=None, last_modified=None)¶

Delete a record in the collection.

Override to perform actions or post-process records after deletion from storage for example:

def delete_record(self, record):
    deleted = super(MyModel, self).delete_record(record)
    erase_media(record)
    deleted['media'] = 0
    return deleted

Parameters:	record (dict) – the record to delete record – record to store parent_id (str) – optional filter for parent id
Returns:	the deleted record.
Return type:	dict

Generators¶

class cliquet.storage.generators.Generator(config=None)¶

Base generator for records ids.

Id generators are used by storage backend during record creation, and at resource level to validate record id in requests paths.

regexp = '^[a-zA-Z0-9][a-zA-Z0-9_-]*$'¶

Default record id pattern. Can be changed to comply with custom ids.

match(record_id)¶

Validate that record ids match the generator. This is used mainly when a record id is picked arbitrarily (e.g with PUT requests).

Returns:	True if the specified record id matches expected format.
Return type:	bool

class cliquet.storage.generators.UUID4(config=None)¶

UUID4 record id generator.

UUID block are separated with -. (example: '472be9ec-26fe-461b-8282-9c4e4b207ab3')

UUIDs are very safe in term of unicity. If 1 billion of UUIDs are generated every second for the next 100 years, the probability of creating just one duplicate would be about 50% (source).

regexp = '^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$'¶

UUID4 accurate pattern.

PostgreSQL¶

class cliquet.storage.postgresql.Storage(client, max_fetch_size, *args, **kwargs)¶

Storage backend using PostgreSQL.

Recommended in production (requires PostgreSQL 9.4 or higher).

Enable in configuration:

cliquet.storage_backend = cliquet.storage.postgresql

Database location URI can be customized:

cliquet.storage_url = postgres://user:pass@db.server.lan:5432/dbname

Alternatively, username and password could also rely on system user ident or even specified in ~/.pgpass (see PostgreSQL documentation).

Note

Some tables and indices are created when cliquet migrate is run. This requires some privileges on the database, or some error will be raised.

Alternatively, the schema can be initialized outside the python application, using the SQL file located in cliquet/storage/postgresql/schema.sql. This allows to distinguish schema manipulation privileges from schema usage.

A connection pool is enabled by default:

cliquet.storage_pool_size = 10
cliquet.storage_maxoverflow = 10
cliquet.storage_max_backlog = -1
cliquet.storage_pool_recycle = -1
cliquet.storage_pool_timeout = 30
cliquet.cache_poolclass = cliquet.storage.postgresql.pool.QueuePoolWithMaxBacklog

The max_backlog limits the number of threads that can be in the queue waiting for a connection. Once this limit has been reached, any further attempts to acquire a connection will be rejected immediately, instead of locking up all threads by keeping them waiting in the queue.

See dedicated section in SQLAlchemy documentation for default values and behaviour.

Note

Using a dedicated connection pool is still recommended to allow load balancing, replication or limit the number of connections used in a multi-process deployment.

Redis¶

class cliquet.storage.redis.Storage(client, *args, **kwargs)¶

Storage backend implementation using Redis.

Warning

Useful for very low server load, but won’t scale since records sorting and filtering are performed in memory.

Enable in configuration:

cliquet.storage_backend = cliquet.storage.redis

(Optional) Instance location URI can be customized:

cliquet.storage_url = redis://localhost:6379/0

A threaded connection pool is enabled by default:

cliquet.storage_pool_size = 50

Memory¶

class cliquet.storage.memory.Storage(*args, **kwargs)¶

Storage backend implementation in memory.

Useful for development or testing purposes, but records are lost after each server restart.

Enable in configuration:

cliquet.storage_backend = cliquet.storage.memory

Exceptions¶

Exceptions raised by storage backend.

exception cliquet.storage.exceptions.BackendError(original=None, message=None, *args, **kwargs)¶

A generic exception raised by storage on error.

Parameters:	original (Exception) – the wrapped exception raised by underlying library.

exception cliquet.storage.exceptions.RecordNotFoundError¶

An exception raised when a specific record could not be found.

exception cliquet.storage.exceptions.UnicityError(field, record, *args, **kwargs)¶

An exception raised on unicity constraint violation.

Raised by storage backend when the creation or the modification of a record violates the unicity constraints defined by the resource.

Public BasicAuth¶

If Basic Auth authentication is enabled, private user resources can become semi-private or public if the user:pass is publicly known and shared (for example public: is a valid user:pass combination). That’s how most simple demos of Kinto — a Cliquet-based application — are built by the way!

BasicAuth trickery¶

Like for user resources, if Basic Auth authentication is enabled, the predictable user id can be used to define semi-private or public if the user:pass is known and shared (for example public: is a valid user:pass combination).

For example, get the user id obtained in the hello root view with a user:pass combination and use it in the permissions JSON payloads, or settings:

cliquet.{collection}_read_principals = basicauth:631c2d625ee5726172cf67c6750de10a3e1a04bcd603bc9ad6d6b196fa8257a6

Related/Inherited permissions¶

In the above section, the list of allowed principals for actions on the collection (especially create) is specified via settings.

It is possible to extend the previously described behavior with related permissions.

For example, in order to imply that having permission to write implies permission to read. Or having permission to create blog articles also means permission to write categories.

To do so, specify the get_bound_permissions of the Cliquet authorization policy.

def get_bound_permissions(self, permission_object_id, permission):
    related = [(permission_object_id, permission)]
    # Grant `read` if user can `write`
    if permission == 'write':
        related.append((permission_object_id, 'read'))
    return related

from pyramid.security import IAuthorizationPolicy

def main(global_settings, **settings):
    ...
    cliquet.initialize(config, __version__)
    ...
    authz = config.registry.queryUtility(IAuthorizationPolicy)
    authz.get_bound_permissions = get_bound_permissions

In Kinto, this is leveraged to implement an inheritance tree of permissions between nested objects. The root objects permissions still have to be specified via settings though.

It is also possible to subclass the default cliquet.authorization.AuthorizationPolicy.

from cliquet import authorization
from pyramid.security import IAuthorizationPolicy
from zope.interface import implementer

@implementer(IAuthorizationPolicy)
class MyAuthz(authorization.AuthorizationPolicy):
    def get_bound_permissions(self, permission_object_id, permission):
        related = [(permission_object_id, permission)]
        # Grant permission on `categories` if permission on `articles`
        if permission_object_id.startswith('/articles'):
            related.append((permission_object_id + '/categories', permission))
        return related