Introduction¶

Dectate is a configuration system that can help you construct Python frameworks. A framework needs to record some information about the functions and classes that the user supplies. We call this process configuration.

Imagine for instance a framework that supports a certain kind of plugins. The user registers each plugin with a decorator:

from framework import plugin

@plugin(name="foo")
def foo_plugin(...):
   ...

Here the framework registers as a plugin the function foo_plugin under the name foo.

You can implement the plugin decorator as follows:

plugins = {}

class plugin(name):
    def __init__(self, name):
        self.name = name

    def __call__(self, f):
       plugins[self.name] = f

In the user application the user makes sure to import all modules that use the plugin decorator. As a result, the plugins dict contains the names as keys and the functions as values. Your framework can then use this information to do whatever you need to do.

There are a lot of examples of code configuration in frameworks. In a web framework for instance the user can declare routes and assemble middleware.

You may be okay constructing a framework with the simple decorator technique described above. But advanced frameworks need a lot more that the basic decorator system described above cannot offer. You may for instance want to allow the user to reuse configuration, override it, do more advanced error checking, and execute configuration in a particular order.

Dectate supports such advanced use cases. It was extracted from the Morepath web framework.

Features¶

Here are some features of Dectate:

• Decorator-based configuration – users declare things by using Python decorators on functions and classes: we call these decorators directives, which issue configuration actions.
• Dectate detects conflicts between configuration actions in user code and reports what pieces of code are in conflict.
• Users can easily reuse and extend configuration: it’s just Python class inheritance.
• Users can easily override configurations in subclasses.
• You can compose configuration actions from other, simpler ones.
• You can control the order in which configuration actions are executed. This is unrelated to where the user uses the directives in code. You do this by declaring dependencies between types of configuration actions, and by grouping configuration actions together.
• You can declare exactly what objects are used by a type of configuration action to register the configuration – different types of actions can use different registries.
• Unlike normal decorators, configuration actions aren’t performed immediately when a module is imported. Instead configuration actions are executed only when the user explicitly commits the configuration. This way, all configuration actions are known when they are performed.
• Dectate-based decorators always return the function or class object that is decorated unchanged, which makes the code more predictable for a Python programmer – the user can use the decorated function or class directly in their Python code, just like any other.
• Dectate-based configuration systems are themselves easily extensible with new directives and registries.

App classes¶

Configuration in Dectate is associated with special classes which derive from dectate.App:

import dectate

class MyApp(dectate.App):
    pass

Creating a directive¶

We can now use the dectate.App.directive() decorator to declare a directive which executes a special configuration action. Let’s replicate the simple plugins example above using Dectate:

@MyApp.directive('plugin')
class PluginAction(dectate.Action):
    config = {
       'plugins': dict
    }
    def __init__(self, name):
        self.name = name

    def identifier(self, plugins):
        return self.name

    def perform(self, obj, plugins):
        plugins[self.name] = obj

Let’s use it now:

@MyApp.plugin('a')
def f():
    pass # do something interesting

@MyApp.plugin('b')
def g():
    pass # something else interesting

We have registered the function f on MyApp. The name argument is 'a'. We’ve registered g under 'b'.

We can now commit the configuration for MyApp:

dectate.commit(MyApp)

Once the commit has successfully completed, we can take a look at the configuration:

>>> sorted(MyApp.config.plugins.items())
[('a', <function f at ...>), ('b', <function g at ...>)]

What are the changes between this and the simple plugins example?

The main difference is that plugin decorator is associated with a class and so its the resulting configuration. The other difference is that we provide an identifier method in the action definition. These differences support configuration reuse, conflicts, extension, overrides and isolation.

class SubApp(MyApp):
   pass

dectate.commit(MyApp, SubApp)

>>> sorted(SubApp.config.plugins.items())
[('a', <function f at ...>), ('b', <function g at ...>)]

class ConflictingApp(MyApp):
    pass

@ConflictingApp.plugin('foo')
def f():
    pass

@ConflictingApp.plugin('foo')
def g():
    pass

>>> dectate.commit(ConflictingApp)
Traceback (most recent call last):
  ...
ConflictError: Conflict between:
 File "...", line 4
   @ConflictingApp.plugin('foo')
 File "...", line 8
   @ConflictingApp.plugin('foo')

def identifier(self, plugins):
    return self.name

@SubApp.plugin('c')
def h():
    pass # do something interesting

dectate.commit(MyApp, SubApp)

>>> sorted(SubApp.config.plugins.items())
[('a', <function f at ...>), ('b', <function g at ...>), ('c', <function h at ...>)]

>>> sorted(MyApp.config.plugins.items())
[('a', <function f at ...>), ('b', <function g at ...>)]

@SubApp.plugin('a')
def x():
    pass

dectate.commit(MyApp, SubApp)

>>> sorted(SubApp.config.plugins.items())
[('a', <function x at ...>), ('b', <function g at ...>), ('c', <function h at ...>)]

>>> sorted(MyApp.config.plugins.items())
[('a', <function f at ...>), ('b', <function g at ...>)]

class BaseApp(dectate.App):
    pass

@BaseApp.directive('plugin')
class PluginAction(dectate.Action):
    config = {
       'plugins': dict
    }
    def __init__(self, name):
        self.name = name

    def identifier(self, plugins):
        return self.name

    def perform(self, obj, plugins):
        plugins[self.name] = obj

class OneApp(BaseApp):
    pass

class TwoApp(BaseApp):
    pass

@OneApp.plugin('a')
def f():
    pass

dectate.commit(OneApp, TwoApp)

>>> sorted(OneApp.config.plugins.items())
[('a', <function f at ...>)]
>>> sorted(TwoApp.config.plugins.items())
[]

The Anatomy of a Directive¶

Let’s consider the directive registration again in detail:

@BaseApp.directive('plugin')
class PluginAction(dectate.Action):
    config = {
       'plugins': dict
    }
    def __init__(self, name):
        self.name = name

    def identifier(self, plugins):
        return self.name

    def perform(self, obj, plugins):
        plugins[self.name] = obj

What is going on here?

• We create a new directive called plugin on MyApp. It also exists for its subclasses.
• The directive is implemented with a custom class called PluginAction that inherits from dectate.Action.
• config (dectate.Action.config) specifies that this directive has a configuration effect on plugins. We declare that plugins is created using the dict factory, so our registry is a plain dictionary. You provide any factory function you like here.
• _init_ specifies the parameters the directive should take and how to store them on the action object. You can use default parameters and such, but otherwise __init__ should be very simple and not do any registration or validation. That logic should be in perform.
• identifier (dectate.Action.identifier()) takes the configuration objects specified by config as keyword arguments. It returns an immutable that is unique for this action. This is used to detect conflicts and determine how configurations override each other.
• perform (dectate.Action.perform()) takes obj, which is the function or class that the decorator is used on, and the arguments specified in config. It should use obj and the information on self to configure the configuration objects. In this case we store obj under the key self.name in the plugins dict.

Once we have declared the directive for our framework we can tell programmers to use it.

Directives have absolutely no effect until commit is called, which we do with dectate.commit. This performs the actions and we can then find the result MyApp.config.

The results are in MyApp.config.plugins as we set this up with config in our PluginAction.

Depends¶

In some cases you want to make sure that one type of directive has been executed before the other – the configuration of the second type of directive depends on the former. You can make sure this happens by using the depends (dectate.Action.depends) class attribute.

First we set up a foo directive that registers into a foos dict:

class DependsApp(dectate.App):
    pass

@DependsApp.directive('foo')
class FooAction(dectate.Action):
    config = {
       'foos': dict
    }
    def __init__(self, name):
        self.name = name

    def identifier(self, foos):
        return self.name

    def perform(self, obj, foos):
        foos[self.name] = obj

Now we create a bar directive that depends on FooDirective and uses information in the foos dict:

@DependsApp.directive('bar')
class BarAction(dectate.Action):
   depends = [FooAction]

   config = {
      'foos': dict,  # also use the foos dict
      'bars': list
   }
   def __init__(self, name):
       self.name = name

   def identifier(self, foos, bars):
       return self.name

   def perform(self, obj, foos, bars):
       in_foo = self.name in foos
       bars.append((self.name, obj, in_foo))

We have now ensured that BarAction actions are performed after FooAction action, no matter what order we use them:

@DependsApp.bar('a')
def f():
    pass

@DependsApp.bar('b')
def g():
    pass

@DependsApp.foo('a')
def x():
    pass

dectate.commit(DependsApp)

We expect in_foo to be True for a but to be False for b:

.. doctest::

>>> DependsApp.config.bars
[('a', <function f at ...>, True), ('b', <function g at ...>, False)]

config dependencies¶

In the example above, the items in bars depend on the items in foos and we’ve implemented this dependency in the perform of BarDirective.

We can instead make the configuration object for the BarDirective depend on foos. This way BarDirective does not need to know about foos. You can declare a dependency between config objects with the factory_arguments attribute of the config factory. Any config object that is created in earlier dependencies of this action, or in the action itself, can be listed in factory_arguments. The key and value in factory_arguments have to match the key and value in config of that earlier action.

First we create an app with a FooAction that sets up a foos config item as before:

class ConfigDependsApp(dectate.App):
    pass

@ConfigDependsApp.directive('foo')
class FooAction(dectate.Action):
    config = {
       'foos': dict
    }
    def __init__(self, name):
        self.name = name

    def identifier(self, foos):
        return self.name

    def perform(self, obj, foos):
        foos[self.name] = obj

Now we create a Bar class that also depends on the foos dict by listing it in factory_arguments:

class Bar(object):
    factory_arguments = {
        'foos': dict
    }

    def __init__(self, foos):
        self.foos = foos
        self.l = []

    def add(self, name, obj):
        in_foo = name in self.foos
        self.l.append((name, obj, in_foo))

We create a BarAction that depends on the FooAction (so that foos is created first) and that uses the Bar factory:

@ConfigDependsApp.directive('bar')
class BarAction(dectate.Action):
   depends = [FooAction]

   config = {
      'bar': Bar
   }

   def __init__(self, name):
       self.name = name

   def identifier(self, bar):
       return self.name

   def perform(self, obj, bar):
       bar.add(self.name, obj)

When we use our directives:

@ConfigDependsApp.bar('a')
def f():
    pass

@ConfigDependsApp.bar('b')
def g():
    pass

@ConfigDependsApp.foo('a')
def x():
    pass

dectate.commit(ConfigDependsApp)

we get the same result as before:

>>> ConfigDependsApp.config.bar.l
[('a', <function f at ...>, True), ('b', <function g at ...>, False)]

before and after¶

It can be useful to do some additional setup just before all actions of a certain type are performed, or just afterwards. You can do this using before (dectate.Action.before()) and after (dectate.Action.after()) static methods on the Action class:

class BeforeAfterApp(dectate.App):
    pass

@BeforeAfterApp.directive('foo')
class FooAction(dectate.Action):
    config = {
       'foos': list
    }
    def __init__(self, name):
        self.name = name

    @staticmethod
    def before(foos):
        print "before:", foos

    @staticmethod
    def after(foos):
        print "after:", foos

    def identifier(self, foos):
        return self.name

    def perform(self, obj, foos):
        foos.append((self.name, obj))

@BeforeAfterApp.foo('a')
def f():
    pass

@BeforeAfterApp.foo('b')
def g():
    pass

This executes before just before a and b are configured, and then executes after:

.. doctest::

>>> dectate.commit(BeforeAfterApp)
before: []
after: [('a', <function f at ...>), ('b', <function g at ...>)]

grouping actions¶

Different actions normally don’t conflict with each other. It can be useful to group different actions together in a group so that they do affect each other. You can do this with the group_class (dectate.Action.group_class) class attribute. Grouped classes share their config and their before and after methods.

class GroupApp(dectate.App):
    pass

@GroupApp.directive('foo')
class FooAction(dectate.Action):
    config = {
       'foos': list
    }
    def __init__(self, name):
        self.name = name

    def identifier(self, foos):
        return self.name

    def perform(self, obj, foos):
        foos.append((self.name, obj))

We now create a BarDirective that groups with FooAction:

@GroupApp.directive('bar')
class BarAction(dectate.Action):
   group_class = FooAction

   def __init__(self, name):
       self.name = name

   def identifier(self, foos):
       return self.name

   def perform(self, obj, foos):
       foos.append((self.name, obj))

It reuses the config from FooAction. This means that foo and bar can be in conflict:

class GroupConflictApp(GroupApp):
    pass

@GroupConflictApp.foo('a')
def f():
    pass

@GroupConflictApp.bar('a')
def g():
    pass

>>> dectate.commit(GroupConflictApp)
Traceback (most recent call last):
  ...
ConflictError: Conflict between:
  File "...", line 8
    @GroupConflictApp.bar('a')

Additional discriminators¶

In some cases an action should conflict with multiple other actions all at once. You can take care of this with the discriminators (dectate.Action.discriminators()) method on your action:

class DiscriminatorsApp(dectate.App):
    pass

@DiscriminatorsApp.directive('foo')
class FooAction(dectate.Action):
    config = {
       'foos': dict
    }
    def __init__(self, name, extras):
        self.name = name
        self.extras = extras

    def identifier(self, foos):
        return self.name

    def discriminators(self, foos):
        return self.extras

    def perform(self, obj, foos):
        foos[self.name] = obj

An action now conflicts with an action of the same name and with any action that is in the extra list:

#

@DiscriminatorsApp.foo('a', ['b', 'c'])
def f():
    pass

@DiscriminatorsApp.foo('b', [])
def g():
    pass

And then:

>>> dectate.commit(DiscriminatorsApp)
Traceback (most recent call last):
  ...
ConflictError: Conflict between:
  File "...", line 3:
    @DiscriminatorsApp.foo('a', ['b', 'c'])
  File "...", line 7
    @DiscriminatorsApp.foo('b', [])

Composite actions¶

When you can define an action entirely in terms of other actions, you can subclass dectate.Composite.

First we define a normal sub directive to use in the composite action later:

class CompositeApp(dectate.App):
    pass

@CompositeApp.directive('sub')
class SubAction(dectate.Action):
    config = {
        'my': list
    }

    def __init__(self, name):
        self.name = name

    def identifier(self, my):
        return self.name

    def perform(self, obj, my):
        my.append((self.name, obj))

Now we can define a special dectate.Composite subclass that uses SubAction in an actions (dectate.Composite.actions()) method:

@CompositeApp.directive('composite')
class CompositeAction(dectate.Composite):
    def __init__(self, names):
        self.names = names

    def actions(self, obj):
        return [(SubAction(name), obj) for name in self.names]

We can now use it:

@CompositeApp.composite(['a', 'b', 'c'])
def f():
    pass

dectate.commit(CompositeApp)

And SubAction is performed three times as a result:

>>> CompositeApp.config.my
[('a', <function f at ...>), ('b', <function f at ...>), ('c', <function f at ...>)]

with statement¶

Sometimes you want to issue a lot of similar actions at once. You can use the with statement to do so with less repetition:

class WithApp(dectate.App):
    pass

@WithApp.directive('foo')
class SubAction(dectate.Action):
    config = {
        'my': list
    }

    def __init__(self, a, b):
        self.a = a
        self.b = b

    def identifier(self, my):
        return (self.a, self.b)

    def perform(self, obj, my):
        my.append((self.a, self.b, obj))

Instead of this:

class VerboseWithApp(WithApp):
    pass

@VerboseWithApp.foo('a', 'x')
def f():
   pass

@VerboseWithApp.foo('a', 'y')
def g():
   pass

@VerboseWithApp.foo('a', 'z')
def h():
   pass

You can instead write:

class SuccinctWithApp(WithApp):
    pass

with SuccinctWithApp.foo('a') as foo:
    @foo('x')
    def f():
        pass

    @foo('y')
    def g():
        pass

    @foo('z')
    def h():
        pass

And this has the same configuration effect:

>>> dectate.commit(VerboseWithApp, SuccinctWithApp)
>>> VerboseWithApp.config.my
[('a', 'x', <function f at ...>), ('a', 'y', <function g at ...>), ('a', 'z', <function h at ...>)]
>>> SuccinctWithApp.config.my
[('a', 'x', <function f at ...>), ('a', 'y', <function g at ...>), ('a', 'z', <function h at ...>)]

importing recursively¶

When you use dectate-based decorators across a package, it can be useful to just import all modules in it at once. This way the user cannot forget to import a module with decorators in it.

Dectate itself does not offer this facility, but you can use the importscan library to do this recursive import. Simply do something like:

import my_package

importscan.scan(my_package, ignore=['.tests'])

This imports every module in my_package, except for the tests sub package.

logging¶

Dectate logs information about the performed actions as debug log messages. By default this goes to the dectate.directive.<directive_name> log. You can use the standard Python logging module function to make this information go to a log file.

If you want to override the name of the log you can set logger_name (dectate.App.logger_name) on the app class:

class MorepathApp(dectate.App):
   logger_name = 'morepath.directive'

Sphinx Extension¶

If you use Sphinx to document your project and you use the sphinx.ext.autodoc extension to document your API, you need to install a Sphinx extension so that directives are documented properly. In your Sphinx conf.py add 'dectate.sphinxext' to the extensions list.

__main__ and conflicts¶

In certain scenarios where you run your code like this:

$ python app.py

and you use __name__ == '__main__' to determine whether the module should run:

if __name__ == '__main__':
    import another_module
    dectate.commit(App)

you might get a ConflictError from Dectate that looks somewhat like this:

Traceback (most recent call last):
 ...
dectate.error.ConflictError: Conflict between:
  File "/path/to/app.py", line 6
    @App.foo(name='a')
  File "app.py", line 6
    @App.foo(name='a')

The same line shows up on both sides of the configuration conflict, but the path is absolute on one side and relative on the other.

This happens because in some scenarios involving __main__, Python imports a module twice (more about this). Dectate refuses to operate in this case until you change your imports so that this doesn’t happen anymore.

How to avoid this scenario? If you use setuptools automatic script creation this problem is avoided entirely.

If you want to use the if __name__ == '__main__' system, keep your main module tiny and just import the main function you want to run from elsewhere.

So, Dectate warns you if you do it wrong, so don’t worry about it.

Install Dectate for development¶

First make sure you have virtualenv installed for Python 2.7.

Now create a new virtualenv somewhere for Dectate’s development:

$ virtualenv /path/to/ve_dectate

The goal of this is to isolate you from any globally installed versions of setuptools, which may be incompatible with the requirements of the buildout tool. You should also be able to recycle an existing virtualenv, but this method guarantees a clean one.

Clone Dectate from github (https://github.com/morepath/dectate) and go to the dectate directory:

$ git clone git@github.com:morepath/dectate.git
$ cd dectate

Now we need to run bootstrap-buildout.py to set up buildout, using the Python from the virtualenv we’ve created before:

$ /path/to/ve_dectate/bin/python bootstrap-buildout.py

This installs buildout, which can now set up the rest of the development environment:

$ bin/buildout

This will download and install various dependencies and tools.

Running the tests¶

You can run the tests using py.test. Buildout will have installed it for you in the bin subdirectory of your project:

$ bin/py.test dectate

To generate test coverage information as HTML do:

$ bin/py.test --cov dectate --cov-report html

You can then point your web browser to the htmlcov/index.html file in the project directory and click on modules to see detailed coverage information.

Running the documentation tests¶

The documentation contains code. To check these code snippets, you can run this code using this command:

$ bin/sphinxpython bin/sphinx-build  -b doctest doc out

Building the HTML documentation¶

To build the HTML documentation (output in doc/build/html), run:

$ bin/sphinxbuilder

Various checking tools¶

The buildout will also have installed flake8, which is a tool that can do various checks for common Python mistakes using pyflakes, check for PEP8 style compliance and can do cyclomatic complexity checking. To do pyflakes and pep8 checking do:

$ bin/flake8 dectate

To also show cyclomatic complexity, use this command:

$ bin/flake8 --max-complexity=10 dectate

0.5 (2016-04-04)¶

• Breaking change The signature of commit has changed. Just pass in one or more arguments you want to commit instead of a list. See #8.

0.4 (2016-04-01)¶

• Expose code_info attribute for action. The path in particular can be useful in implementing a directive such as Morepath’s template_directory. Expose it for composite too.
• Report a few more errors; you cannot use config, before or after after in an action class if group_class is set.
• Raise a DirectiveReportError if a DirectiveError is raised in a composite actions method.

0.3 (2016-03-30)¶

• Document importscan package that can be used in combination with this one.
• Introduced factory_arguments feature on config factories, which can be used to create dependency relationships between configuration.
• Fix a bug where config items were not always properly reused. Now only the first one in the action class dependency order is used, and it is not recreated.

0.2 (2016-03-29)¶

• Remove clear_autocommit as it was useless during testing anyway. In tests just use explicit commit.
• Add a dectate.sphinxext module that can be plugged into Sphinx so that directives are documented properly.
• Document how Dectate deals with double imports.

0.1 (2016-03-29)¶

• Initial public release.