====================
Apycot Quick Start
====================
--------
Overview
--------
Apycot is designed to be an extensible test automation tool usable for
continuous integration and continuous testing. It is a CubicWeb-based
application extending the Narval framework. It adds the notion of preprocessors
and checkers to Narval. The preprocessors are used mainly to checkout
version-controlled repositories and setup a test environment while the checkers
correspond to specific checks (pycoverage, pytest, pylint...).
Other major differences with respect to Narval include the use of TestExecution
and CheckResults instead of Plans to store the results and the organisation of
test environments through ProjectEnvironments and TestConfigs.
The Apycot framework, like Narval is split between:
- a Web application used for storing, presenting and accessing the test results;
- a bot regularly polling the web application using http, running the tests
when necessary and transmitting back the results to the Web application.
Both parts can be installed on the same machine or on different machines able
to communicate via HTTP. Please read the Narval documentation before installing
Apycot.
-----------
Definitions
-----------
* A **repository** is a version-controlled sources (VCS) database, like SVN or
Hg.
* A **preprocessor** allows a specific construction / installation step
required to build a test environment. Example preprocessors: 'setup_install'
to install a package by using `python setup.py install`, 'make' call the
`make` command...
* A **check** is a single functional test which may be applied to test a
package. A **checker** is the object applying this functional test. Example
checks: 'pyunit' to start python unittest, 'pylint' to start pylint...
---------
Narvalbot
---------
Installation
============
In addition to what is described in the Narval documentation, you need to install
the additional ``narval-apycot`` package (it contains the additional checkers and
preprocessors used by apycot).
.. sourcecode:: bash
apt-get install narval-bot narval-apycot
Configuration
=============
Follow the Narval bot configuration documentation to configure the Narval bot.
---------------
Cubicweb-apycot
---------------
Installation
============
To install a complete Apycot environment, you need to install the following packages:
.. sourcecode:: bash
apt-get install cubicweb-apycot cubicweb-ctl
Creating an Apycot instance
===========================
.. sourcecode:: bash
cubicweb-ctl create apycot myapycot
You can then launch the CubicWeb instance:
.. sourcecode:: bash
cubicweb-ctl start -D myapycot
Make sure the bot manages to connect to the instance by looking at the logs in `/var/log/narval/narval.log`. If you see something like::
2014-01-15 20:20:40 - (narval.bot) INFO: get pending plan from narval
2014-01-15 20:20:40 - (requests.packages.urllib3.connectionpool) INFO: Starting new HTTP connection (1): crater2.logilab.fr
it means everything is working as expected. If the Narval bot cannot connect, you will see this instead::
Traceback (most recent call last):
File "/usr/lib/pymodules/python2.7/narvalbot/server.py", line 107, in _loop
for plandata in self.config.cnxh(instance_id).pending_plans():
File "/usr/lib/pymodules/python2.7/narvalbot/__init__.py", line 222, in pending_plans
'?vid=narval.pending-plans'))
File "/usr/lib/pymodules/python2.7/narvalbot/__init__.py", line 182, in http_get
return self.http_post(url, method='get')
File "/usr/lib/pymodules/python2.7/narvalbot/__init__.py", line 192, in http_post
self.connect()
File "/usr/lib/pymodules/python2.7/narvalbot/__init__.py", line 212, in connect
'?__login=%s&__password=%s' % (user, password)))
File "/usr/lib/pymodules/python2.7/narvalbot/__init__.py", line 247, in open
return self.opener.open(*args, **kwargs)
File "/usr/lib/python2.7/urllib2.py", line 401, in open
response = self._open(req, data)
File "/usr/lib/python2.7/urllib2.py", line 419, in _open
'_open', req)
File "/usr/lib/python2.7/urllib2.py", line 379, in _call_chain
result = func(*args)
File "/usr/lib/python2.7/urllib2.py", line 1211, in http_open
return self.do_open(httplib.HTTPConnection, req)
File "/usr/lib/python2.7/urllib2.py", line 1181, in do_open
raise URLError(err)
URLError: <urlopen error [Errno 111] Connection refused>
Setting up a test environment
=============================
We first describe the entity types added by Apycot and then provide a complete
setup procedure.
Creating a project environment (ProjectEnvironment)
---------------------------------------------------
To setup a ProjectEnvironment, you need to specify:
- the ProjectEnvironment name
- the configuration variables. These variables are available in the
checkers and preprocessors in the options object. For example ::
install=python_setup
pycoverage=True
extra_argument=['-m', 'Corp']
test_dirs=['tests']
test_prefixes=['test_','unittest_']
pylint_threshold=70
pylintrc='~/.pylintrc'
coverage_threshold=0.7
keep_test_dir=True
archive=False
required=False
verbose=True
- the environment variables. These variables will be available in the
shell for the preprocessors and checkers. For example::
NO_SETUPTOOLS=1
DISPLAY=:1.0
Usually, when checkers or preprocessors fail, or crash, some of these variables
are incorrect or missing.
Creating a test configuration (TestConfig)
------------------------------------------
Usually, a recipe requires at least a TestConfig. This configuration
contains all the necessary information to run the tests.
To setup a TestConfig, you need to specify:
- the TestConfig name
- the launching mode ("manual" by default)
- the way dependent project environments are tested when the TestConfig
is modified; can be "yes" (thus you lauch the tests for these project
environments), "no" (thus, you dont lauch the tests), and "inherited" (the
value is set elsewhere).
- the configuration variables and environment variables. These are similar to
the configuration variables found in ProjectEnvironment instances.
- the recipe associated to the TestConfig. This recipe will be launched
by the narval instance when this configuration will be used.
- the "refinement of" specifies the parent TestConfig. The current TestConfig
will inherit all the attributes and variables from the parent TestConfig.
To run tests you must link the TestConfig to a ProjectEnvironment and specify a
recipe. Make sure the options necessary for each checker and preprocessor are
availalbe in either the ProjectEnvironment or the TestConfig.
Creating a repository
---------------------
The Repository is a proxy to a VCS repository holding the project to be tested,
e.g. Mercurial.
To setup a Repository, you need to specify:
- the type of the VCS handling the Repository;
- the path to the Repository.
Make sure the Repository can be accessed by both the Apycot Web application and
the Narval bot instance.
In the "files" tab you can find all the files in the current revision, whereas
in the "revisions" tabs you have a table with all revisions known to the
Repository.
Full setup procedure
--------------------
Make sure the cubicweb instance running Apycot and the Narval bot instance are
both installed, configured (as shown above and in the Narval documentation) and
able to communicate (see their respective install procedures).
1. Setup a Repository for the project
2. Create a TestConfig for each recipe you want to be able to run.
- make sure the correct recipe is selected for each TestConfig.
- make sure to fill-in the necessary options (``install=python_setup`` is
usually mandatory for Python recipes).
3. Setup a ProjectEnvironment for the project
- link the ProjectEnvironment to the previously created Repository
- link the ProjectEnvironment to all the necessary TestConfig instances.
4. From the ProjectEnvironment view in cubicweb, manually launch the tests and
watch the results appear on you display (if you do not see the option to
launch tests, make sure you are logged in and have the necessary
authorizations).
Troubleshooting
---------------
The Narval bot never gets the plan and the interval between the bot's polls is
too long:
- the interval can be modified in `/etc/narval/narval-cw-sources.ini` to add
the ``poll-delay`` argument::
[narval]
url=http://instance-name.fr:10003/
token_id=token name
secret=generated secret
poll-delay=5
The checkers crash but the project tests are fine, the project is not installed
correctly:
- make sure no configuration and environment variables are missing or wrongly
set.
- check the paths used by Narval bot are writable and free space is available
on the disk.