readme.rst
author Arthur Lutz <arthur.lutz@logilab.fr>
Thu, 25 Apr 2013 16:15:53 +0200
changeset 146 98bf5201c24d
parent 145 ff7a935c3316
child 147 912e8362c012
permissions -rw-r--r--
[readme] CentOS specific instructions


Objective
=========

Automatically create variations of virtual machines images for a cloud
infrastructure (OpensStack or Amazon are currently supported) using
SaltStack_ based configuration files availables in a given Mercurial_
repository, eg.:

  http://hg.logilab.org/users/ptonelli/ami_creator/variation

The AMI creator tool consist in a properly configured AMI to be
instanciated with proper parameters passed as user data. This instance
(which creation itself is described below) by default also updates
itself from the base configuration repository (stored in the following
Mercurial_ repository):

  http://hg.logilab.org/users/ptonelli/ami_creator/server

.. _SaltStack: http://docs.saltstack.com/
.. _Mercurial: http://mercurial.selenic.com/

How to create an AMI
====================

Creating an AMI is as simple as instanciating the ami-creator image in
your cloud infrastructure with proper user-data content.

What will be installed and configured in the built AMI image will
depend on the configuration description that will be used as SaltStack
highstate_.

.. _highstate: http://docs.saltstack.com/ref/states/highstate.html

So to summarize, the process of building a customized AMI image is:

1. Edit/commit/push the file "variation.sls" and all its dependencies
   in a Mercurial repository (must be accessible from the cloud
   instance) so that a:

.. sourcecode:: bash
     user@host:~$ salt-call state.sls variation.sls

   will create the machine you want.

2. Then, launch an instance of the "ami-creator" image in your
   cloud. Use at least 5Gb of disk (add more if your sls file requires
   more).

   In the userdata form of your instance, copy something like (if you
   do not put anything, what will be used will be the latest revision
   from the default repo).

   Warning : the file needs to un-indented (remove whitespaces at beginning of lines) ::

     ====== START =======
     #salt-grains #interpreter flag
     ### necessary unless you want the standard image ###
     ami.hg_address_variation: http://hg.logilab.org/users/ptonelli/ami_creator/variation
     ami.hg_rev_variation: 9ed741a55f17 #optionnal
     ami.name: my_variation
     ami.type: ubuntu #or debian

     ami.source_address: http://cloud-images.ubuntu.com/precise/current/precise-server-cloudimg-amd64.tar.gz
     ami.source_hash: md5=1d72ed9c56abb899be02e7cae0822f1d

     ### optionnal if your image uses existing kernel/ramdisk id in openstack ###
     kernel_id:a5b900bd-c009-40b2-a763-32b32996b1ee
     initrd_id:b953e40a-2605-402b-8663-c8556a5899f2

     ### optionnal (ami_creation code update) ###
     ami.hg_address_server: http://hg.logilab.org/users/ptonelli/ami_creator/server

     ### optionnal (necessary to upload to openstack) ###
     keystone.user: username
     keystone.password: password
     keystone.tenant: tenant
     keystone.tenant_id: 00000000000000000000000000000000
     keystone.auth_url: 'http://control.example.com:5000/v2.0'
     keystone.insecure: False   #(optional)

     ### optionnal (necessary to upload to amazon aws) ###
     ### remove the BEGIN and END statements found in the files ###
     ### do not forget the ' ' at each newline for ConfigParser ###
     aws.region: eu-west-1
     aws.private_key: 0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
     aws.certificate: 0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000

     ====== END ========

3. you can then connect to your instance and wait until everything in ``/mnt``
   is unmounted. Upload to glance (openstack) or amazon aws is automatic if you
   provided the correct user data. 

Main Files
----------

minion: must be copied to /etc/salt, provides a masterless configuration for salt


ami_creator.sls: state file containing all operations to create the new image 

ami_pre: state folder (contains a few sls files) applied before the creation of variation

ami_post: state folder applied after the creation of variation for
cleanup

Additionnal Documentation
=========================

All the operations are currently done in /etc/rc.local. The content of the file
is explained in the following lines.

The first step parses the user-data related to our ami

::
  /usr/bin/env python /etc/salt/srv/others/ec2_ami_parser.py

The next step updates the salt states, if necessary by getting the necessary
files from the hg repository

::
  salt-call state.highstate
  salt-call state.highstate

(doing it twice is necessary to load new state/modules from the hg repository). Next,

::
  salt-call state.sls ami_creator

includes all the operations done in the image. The operations can be grouped in three steps:

    1. preparation (ami_pre): download the image, uncompress it, resize the fs,
       mount it, modify the relevant files to work in a chroot
    2. update the image, download and execute the variation files from the mercurial variation URL
    3. postprocessing (ami_post): unmount, resize and upload the image either
       to amazon or to openstack depending on user_data

Bootstrapping an ami_creator machine
=====================================

This should only be used if no machine is available. If an ami_creator image is
already available, juste launch it with the necessary user data:

    1. ami.type: ubuntu
    2. a correct source/md5sum for an ubuntu image
    3. use the server repository as variation address.
    4. provide your correct credentials

This should create and upload an up to date custom AMI image

On Ubuntu, the first step is to add the salt PPA to install it:

.. sourcecode:: bash
  root@machine~# add-apt-repository -y ppa:saltstack/salt

The next step is to install salt-minion and mercurial:

.. sourcecode:: bash
  root@machine~# aptitude update && aptitude install salt-minion mercurial

On CentOS, enable EPEL and install salt-minon & mercurial : 

.. sourcecode:: bash
  root@machine~# wget http://dl.fedoraproject.org/pub/epel/5/x86_64/epel-release-5-4.noarch.rpm
  root@machine~# rpm -Uvh epel-release-5-4.noarch.rpm
  root@machine~# yum install salt-minion mercurial


Then, clone the server repository:

.. sourcecode:: bash
  root@machine~# hg clone http://hg.logilab.org/users/ptonelli/ami_creator/server/ /etc/salt/srv

Copy the necessary scripts so the ami creation process is launched at boot:

.. sourcecode:: bash
  root@machine~# cp /etc/salt/srv/others/rc.local /etc/rc.local

and copy the configuration to make salt-minion masterless:

.. sourcecode:: bash
  root@machine~# cp /etc/salt/srv/minion /etc/salt/

save this machine as an ami-creator image and you can then use it by simply
adding relevant user data at boot

Manual upload to openstack (glance)
===================================

This procedure should not be necessary, the openstack client is functionnal and
will be run, provided you include the relevant user data.

After the end of the creation process (nothing is mounted in /mnt/variation and
you can see a .img, a kernel and a ramdisk in /mnt/output), you can upload your
image to glance manually, provided you have credentials as a openrc.sh file
downloaded from openstack.

1. First, add your credentials to path (change ):

.. sourcecode:: bash  
  root@machine~# . /${correct_path}/openrc.sh 

2. Then, if necessary, define a name for your ami:

.. sourcecode:: bash
  root@machine~# export ami_name=variation

3. And change to the correct directory:

.. sourcecode:: bash
  root@machine~# cd /mnt/output 

4. If necessary, upload the kernel and the ramdisk:

.. sourcecode:: bash
  root@machine~# glance add is_public=false disk_format=aki container_format=aki \
  name=${ami_name}_kernel < kernel_variation

.. sourcecode:: bash
  root@machine~# glance add is_public=false disk_format=ari container_format=ari \
  name=${ami_name}_ramdisk < ramdisk_variation

5. Finally, upload the image after replacing the kernel_id and ramdisk_id
   according to the previous uploads:

.. sourcecode:: bash
  root@machine~# glance add disk_format=ami container_format=ami name="${ami_name}" \
  kernel_id=278de93c-5276-410f-a221-667cc1443cf8 \
  ramdisk_id=b953e40a-2605-402b-8663-c8556a5899f2 < /mnt/variation.img

Manual upload to ec2
====================

This procedure should not be necessary, the amazon ec2 client is functionnal and
will be run, provided you include the relevant user data.

How to upload the ami you created on amazon manually (from_):

1. Create an EBS volume of the minimal size to host the disk image

2. attach the created volume to ami_creator: 

3. transfer the content of the image to the EBS volume (replace xvdf by correct
   device name):

.. sourcecode:: bash
  ubuntu@machine:~$ sudo dd bs=16M if=/mnt/output/variation.img of=/dev/xvdf

4. Detach the volume, take a snapshot, create an ami image from the amazon ec2
   web interface. Make sure to use the correct "aki" (kernel image) as
   indicated on documentation_ (for eu, use aki-71665e05).

::_from: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_LaunchingInstanceFromSnapshot.html
::_documentation: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/UserProvidedkernels.html