CWEP-004.rst
author Christophe de Vienne <christophe@unlish.com>
Wed, 22 Oct 2014 20:24:55 +0200
changeset 19 d06afae95fe4
permissions -rw-r--r--
CWEP 0004 initial draft

=============================
 CWEP-0004 Cubes as packages
=============================

:champion: Christophe de Vienne
:reviewers: COPIL
:last update: 2014/10/22
:status: draft

Abstract
========

This document specifies a proposed standard for cubes layout, installation and
discovery.

The central idea is to use standard python packages as containers, and rely on
entry-points to discover the cubes.

Rationale
=========

CubicWeb currently handle the cubes in its own way, without clear advantage
over user standard python packages.

This situation leads to several problematic situations, with no easy solution:

-   If modules are imported from one cube to another, multiple values in
    CUBES_PATH is not possible, because the root "cubes" directory is expected
    to be in the PYTHONPATH.

    This particular issue makes it difficult, if not impossible, to install
    cubes system-wide (or in a virtualenv) and work on cubes in a work
    directory.

-   The (unperfect but popular) "develop" mode of setuptools and pip does not
    work. It can be worked around but it is a pain (See
    http://lists.cubicweb.org/pipermail/cubicweb/2014-September/002160.html)

-   The schema of a cube can only be in a "schema.py" file, not a "schema"
    module.

-   The code handling the cubes is complex.

-   The dependencies between cubes are only used to enforce dependencies at
    runtime, but not to install them (except if using the debian packages).

-   The implicit scan and registering of objects depending on the module
    they belong to is a source of confusion, and a limition of possible
    layouts.

The design choices for the current way cubes are handled made sens when it was
conceived.

Since then however, the python packaging went throught a lot of rationalization
and standardization, and now provides all CubicWeb needs to define a simpler
way to create cubes.

The reading of the `Python Packaging User Guide`_ is recommended to get an
updated summary of the python packaging ecosystem

Specifications
==============

Cube
----

What is a cube ?
~~~~~~~~~~~~~~~~

A cube has a name, that should be unique in the cubes ecosystem.

A cube has a version, that can be used to define requirements.

A cube has a schema version, that is used to detect if a the database needs an
upgrade. In the very large majority of cases, this version is the same as the
cube version.

A cube provides:

Things loaded at cube loading time:

-   Properties (uiprops, site_cubicweb)

Things loaded at registration time (almost the same as loading time):

-   Registrable objects (mostly AppObjects)
-   Static resources (indirectly)

Things loaded when schema definition is needed, mainly to initialize a
migration context:

-   Schema

Things loaded exceptionnally, ie when a migration is needed:

-   Migration scripts

Cube Interface
~~~~~~~~~~~~~~

The Cube interface is what it is expected to provide so that CubicWeb is able
to find it and load what is inside.

Currently, the cube interface is a list of file and directory with precise
names, allowing little, if any, customization.

The proposed new interface consists of a few functions/objects:

options
    The specific configuration options of the cube. Strictly equivalent to
    site_cubicweb.options variable.


def register(vreg):
    Use the vreg methods to:
    
    - scan module(s) for buildobjs (ie 'schema')
    - scan module(s) for app objects.
    - declare a directory containing static resources
    - declare a directory containing the migration scripts.
    - declare / alter uiprops


def includeme(config)
    Called in the context of a pyramid based cubicweb instance.
    See `Pyramid Advanced Configuration`_ for more information on the expected
    behavior of this function.
    
    This particular function will always be called _after_ init_cube.


Importing these function should have minimal side-effect, as importing them
does not mean activating them.

Compatible Cubes
~~~~~~~~~~~~~~~~

The new interface being compatible with the old one, we can have cube sources
that can produce both an old-fashion cube (installed in the cubes path), and
new-fashion cubes (installed as packages).

These cubes will need a special option on the "setup.py install" to be
installed the old way.

Cubes lookup
------------

Any python package declaring any entry-point in the "cubicweb" group is
considered a cube.

`pkg_resources` provides a `convenience API`_ to lookup all entry-points
in a group.

A sample code for listing available cubes could be:

.. code-block:: python

    def list_cubes():
        dists = set()
        for entrypoint in pkg_resource.iter_entry_points('cubicweb'):
            dists.append(entrypoint.dist)
        return [dist.name for dist in dists]

.. _convenience API: https://pythonhosted.org/setuptools/pkg_resources.html#convenience-api

Migration Path
--------------

To avoid forcing all cubes to be converted at once, we need to set up a smooth
path.

Phase 1
-------

Proposed target: Cubicweb 3.20

At this stage, the new cubes interfaces and apis are not frozen, hence people
using them must be ready to fix their cubes between two cubicweb releases.

Phase 2
-------

Proposed target: Cubicweb 3.21

The new cubes api is frozen.

The old fashion cubes are still loaded, but with deprecation warnings.

During this phase, cube maintainers should make their cubes compatible.

Phase 3
-------

Proposed target: Cubicweb 4.0

Old-fashion cubes are not loaded anymore.

Conclusion
==========


.. _Python Packaging User Guide: https://python-packaging-user-guide.readthedocs.org/en/latest/current.html
.. _Pyramid Advanced Configuration: http://pyramid.readthedocs.org/en/latest/narr/advconfig.html?highlight=includeme#including-configuration-from-external-sources