improve the readme file (closes #210062 ):
authorPaul Tonelli <paul.tonelli@logilab.fr>
Wed, 27 Mar 2013 19:19:59 +0100
changeset 110 9cabb9379a03
parent 109 3b7c74c614ca
child 111 a881ea816c4e
improve the readme file (closes #210062 ): - general syntax - added a ami-creator creator to bootstrap the process - update the files (only talk about the main files) - update the user data flags
readme.rst
--- a/readme.rst	Wed Mar 27 10:37:34 2013 +0100
+++ b/readme.rst	Wed Mar 27 19:19:59 2013 +0100
@@ -2,174 +2,251 @@
 Objective
 =========
 
-Automatically create variations of virtual machines depending
-on configuration files availables in repository :
+Automatically create variations of virtual machines images for a cloud
+infrastructure (OpensStack or Amazon are currently supported) using
+SaltStack_ based configuration files availables in a given Mercurial_
+repository, eg.:
 
   http://hg.logilab.org/users/ptonelli/ami_creator/variation
 
-
-it also automatically gets the correct scripts from while doing so: 
+The AMI creator tool consist in a properly configured AMI to be
+instanciated with proper parameters passed as user data. This instance
+(which creation itself is described below) by default also updates
+itself from the base configuration repository (stored in the following
+Mercurial_ repository):
 
   http://hg.logilab.org/users/ptonelli/ami_creator/server
 
-Present state
-=============
-
+.. _SaltStack: http://docs.saltstack.com/
+.. _Mercurial: http://mercurial.selenic.com/
 
-The first step is to edit/commit/push the file "variation.sls" so that a:
-
-salt-call state.sls variation.sls 
+How to create an AMI
+====================
 
-will create the machine you want.
+Creating an AMI is as simple as instanciating the ami-creator image in
+your cloud infrastructure with proper user-data content.
 
-You can also clone the repository and commit to another hg repo at the
-condition that this repository can be reached by http / https.
+What will be installed and configured in the built AMI image will
+depend on the configuration description that will be used as SaltStack
+highstate_.
 
-Then, launch an instance of "ami-creation" on openstack. Use at
-least 5Gb of disk (add more if your sls file requires more).
+.. _highstate: http://docs.saltstack.com/ref/states/highstate.html
+
+So to summarize, the process of building a customized AMI image is:
 
-In the userdata, copy something like (if you do not put anything, what will be
-used will be the latest revision from the default repo):
+1. Edit/commit/push the file "variation.sls" and all its dependencies
+   in a Mercurial repository (must be accessible from the cloud
+   instance) so that a:
 
-====== START =======
+.. sourcecode:: bash
+     user@host:~$ salt-call state.sls variation.sls
 
-ami.hg_rev_variation: 249:9ed741a55f17
-ami.hg_address_variation: http://hg.logilab.org/users/ptonelli/ami_creator/variation
-ami.name: my_variation 
-ami.source_address: http://cloud-images.ubuntu.com/precise/current/precise-server-cloudimg-amd64.tar.gz
-ami.type: debian #or ubuntu
+   will create the machine you want.
 
-### optionnal ###
-keystone.user: usearname
-keystone.password: password
-keystone.tenant: tenant
-keystone.tenant_id: 00000000000000000000000000000000
-keystone.auth_url: 'http://control.example.com:5000/v2.0'
-keystone.insecure: False   #(optional)
+2. Then, launch an instance of the "ami-creator" image in your
+   cloud. Use at least 5Gb of disk (add more if your sls file requires
+   more).
+
+   In the userdata form of your instance, copy something like (if you
+   do not put anything, what will be used will be the latest revision
+   from the default repo)::
 
-### only if you know what you are doing ! ###
-kernel_id:a5b900bd-c009-40b2-a763-32b32996b1ee
-initrd_id:b953e40a-2605-402b-8663-c8556a5899f2
-
-====== END ========
-
-The kernel_id and initrd_id are used to specify the 
+     ====== START =======
+     ## salt-grains #interpreter flag
+     ### necessary unless you want the standard image ###
+     ami.hg_address_variation: http://hg.logilab.org/users/ptonelli/ami_creator/variation
+     ami.hg_rev_variation: 9ed741a55f17 #optionnal
+     ami.name: my_variation
+     ami.type: ubuntu #or debian
 
-you can add an ip to your instance, ssh to the machine and wait until
-everything in /mnt is unmounted. you can then upload what you want to glance
-with something like (don't forget to change the ids in the last tim or to
-include your credentials by downloading and running openrc.sh):
+     ami.source_address: http://cloud-images.ubuntu.com/precise/current/precise-server-cloudimg-amd64.tar.gz
+     ami.source_hash: md5=1d72ed9c56abb899be02e7cae0822f1d
 
-glance add is_public=false disk_format=aki container_format=aki \
-name=variation_kernel < kernel_variation
+     ### optionnal if your image uses existing kernel/ramdisk id in openstack ###
+     kernel_id:a5b900bd-c009-40b2-a763-32b32996b1ee
+     initrd_id:b953e40a-2605-402b-8663-c8556a5899f2
+
+     ### optionnal (ami_creation code update) ###
+     ami.hg_address_server: http://hg.logilab.org/users/ptonelli/ami_creator/server
 
-glance add is_public=false disk_format=ari container_format=ari \
-name=variation_ramdisk < ramdisk_variation
+     ### optionnal (necessary to upload to openstack) ###
+     keystone.user: usearname
+     keystone.password: password
+     keystone.tenant: tenant
+     keystone.tenant_id: 00000000000000000000000000000000
+     keystone.auth_url: 'http://control.example.com:5000/v2.0'
+     keystone.insecure: False   #(optional)
 
-glance add disk_format=ami container_format=ami name="variation-amd64" \
-kernel_id=278de93c-5276-410f-a221-667cc1443cf8 \
-ramdisk_id=b953e40a-2605-402b-8663-c8556a5899f2 < /mnt/variation.img
-
+     ====== END ========
 
-Files
------
+3. you can then connect to your instance and wait until everything in ``/mnt``
+   is unmounted. Upload to glance (openstack) is automatic if you provided the
+   correct user data. If you want to upload to amazon ec2, you need to follow
+   the following steps: 
 
-top.sls : top file for the ami_creator
+.. sourcecode:: bash
+     root@amibuilder:~#  
 
-ami_creator.sls : state file to have a correct ami-builder environment
+Main Files
+----------
+
+minion: must be copied to /etc/salt, provides a masterless configuration for salt
 
-ami_creator_pre.sls : state file applied before the creation of variation
+ami_pre: state folder (contains a few sls files) applied before the creation of variation
 
-ami_creator_post.sls : state file applied after the creation of variation for
+ami_creator.sls: state file to have a correct ami-builder environment
+
+ami_post: state folder applied after the creation of variation for
 cleanup
 
-variation.sls : variation file (should be modified depending on requirements)
-
-variation.top.sls : top file for the chroot
-
+Additionnal Documentation
+=========================
 
-Additionnal Documentation
-========================
+All the operations are currently done in /etc/rc.local. The content of the file
+is explained in the following lines.
 
-what is being done and where :
+The first step parses the user-data related to our ami
 
-in /etc/init.d/ec2-run-ami
---------------------------
+::
+  /usr/bin/env python /etc/salt/srv/others/ec2_ami_parser.py
 
-If the userdata is not a script and is not compressed content, Add the content
-of userdata as salt-grains (version and address from repo...)
+The next step updates the salt states, if necessary by getting the necessary
+files from the hg repository
 
-in /etc/rc.local
-----------------
-
+::
   salt-call state.highstate
   salt-call state.highstate
 
-(doing it twice is necessary to load new states)
-
-  salt-call vm_create.create_variation 
+(doing it twice is necessary to load new state/modules from the hg repository). Next,
 
-which contains : 
+::
+  salt-call state.sls ami_pre
 
-  salt-call state.sls ami_creator_pre
-
-which mounts a debian image and creates all the necessary files inside. Then
-update the chroot:
+downloads and mounts an image and creates all the necessary files inside.
 
-  salt-call vm_create.update_image /mnt/variation
+::
+  salt-call state.sls ami_creator
 
-and run your state file (if necessary, modify /etc/salt/srv/variation.sls which
-will be executed inside):
-
-  salt-call vm_create.variate_image /mnt/variation
+updates the chroot, install salt and run your state file. Finally
 
-which runs your variation inside the chroot. Finally run
+::
+  salt-call state.sls ami_post
 
-  salt-call state.sls ami_creator_post
+extracts your image kernel and ramdisk and unmount and resize as necessary
 
-which will extract your image kernel and ramdisk and unmount and resize as necessary
-
+::
   salt-call vm_create.upload to glance
 
-if the correct userdata is available, this sends the kernel, ramdisk and image
-to glance using the glance python client.
+if the correct userdata is available, it also uploads the kernel, ramdisk and image
+to glance using the glance python client. Uploading to amazon ec2 is not yet functionnal
+
+TODO
+====
+
+    1. add an amazon ec2 client to :
+
+        a. create and mount an EBS volume
+
+        b. copy the contente of the img file to this volume
 
-glance
-------
+        c. unmount and detach the volume 
+
+        d. take a snapshot of the volume
+
+        e. register the snapshot as an ami in amazon
 
-The salt module from glance does not work in debian as ``glanceclient`` package
-is only available in experimental and I did not manage to connect to openstack
-essex with it.
+    2. include the bootstrap code as a variation sls file
+
+Bootstrapping an ami_creator machine
+=====================================
+
+The first step is to add the salt ppa to install it:
 
-nova
-----
+.. sourcecode:: bash
+  root@machine~# add-apt-repository -y ppa:saltstack/salt
+
+The next step is to install salt-minion and mercurial:
 
-The salt module is working as intended.
+.. sourcecode:: bash
+  root@machine~# aptitude update && aptitude install salt-minion mercurial
+
+Then, clone the server repository:
+
+.. sourcecode:: bash
+  root@machine~# hg clone http://hg.logilab.org/users/ptonelli/ami_creator/server/ /etc/salt/srv
 
-Optimal behavior of this project
-================================
+Copy the necessary scripts to have launch the ami creation process at boot:
+
+.. sourcecode:: bash
+  root@machine~# cp /etc/salt/srv/others/rc.local /etc/rc.local
+
+and copy the configuration to make salt-minion masterless:
 
-3 levels of scripting:
+.. sourcecode:: bash
+  root@machine~# cp /etc/salt/srv/minion /etc/salt/
+
+save this machine as an ami-creator image and you can then use it by simply
+adding relevant user data at boot
+
+Manual upload to openstack (glance)
+===================================
 
-Virtual Instance Launcher
--------------------------
+After the end of the creation process (nothing is mounted in /mnt/variation and
+you can see a .img, a kernel and a ramdisk in /mnt/output), you can upload your
+image to glance manually, provided you have credentials as a openrc.sh file
+downloaded from openstack.
+
+1. First, add your credentials to path (change ):
+
+.. sourcecode:: bash  
+  root@machine~# . /${correct_path}/openrc.sh 
+
+2. Then, if necessary, define a name for your ami:
+
+.. sourcecode:: bash
+  root@machine~# export ami_name=variation
 
-(hooked to modifications of HG repository)
-NOTHING DONE
+3. And change to the correct directory:
+
+.. sourcecode:: bash
+  root@machine~# cd /mnt/output 
+
+4. If necessary, upload the kernel and the ramdisk:
 
-Ami Creator
------------
+.. sourcecode:: bash
+  root@machine~# glance add is_public=false disk_format=aki container_format=aki \
+  name=${ami_name}_kernel < kernel_variation
 
-An image available on openstack / ec2.
+.. sourcecode:: bash
+  root@machine~# glance add is_public=false disk_format=ari container_format=ari \
+  name=${ami_name}_ramdisk < ramdisk_variation
 
-At boot, the user must supply a repository address in user data as
-``variation_address`` this information is used by salt at boot to download the
-correct revision of files.  it then execute salt-call
-vm_create.create_variation which will create a complete ami file and upload it
-to glance EC2
+5. Finally, upload the image after replacing the kernel_id and ramdisk_id
+   according to the previous uploads:
+
+.. sourcecode:: bash
+  root@machine~# glance add disk_format=ami container_format=ami name="${ami_name}" \
+  kernel_id=278de93c-5276-410f-a221-667cc1443cf8 \
+  ramdisk_id=b953e40a-2605-402b-8663-c8556a5899f2 < /mnt/variation.img
+
+Manual upload to ec2
+====================
+
+How to upload the ami you created on amazon manually (from_):
+
+1. Create an EBS volume of the minimal size to host the disk image
 
-Chroot Image
-------------
+2. attach the created volume to ami_creator: 
+
+3. transfer the content of the image to the EBS volume (replace xvdf by correct
+   device name):
 
-this script should only be a salt state file, and must be run using
-salt-highstate before packaging of the ami.
+.. sourcecode:: bash
+  ubuntu@machine:~$ sudo dd bs=16M if=/mnt/output/variation.img of=/dev/xvdf
+
+4. Detach the volume, take a snapshot, create an ami image from the amazon ec2
+   web interface. Make sure to use the correct "aki" (kernel image) as
+   indicated on documentation_ (for eu, use aki-71665e05).
+
+::_from: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_LaunchingInstanceFromSnapshot.html
+::_documentation: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/UserProvidedkernels.html