improve the readme file (closes #210062 ):
authorPaul Tonelli <>
Wed, 27 Mar 2013 19:19:59 +0100
changeset 110 9cabb9379a03
parent 109 3b7c74c614ca
child 111 a881ea816c4e
improve the readme file (closes #210062 ): - general syntax - added a ami-creator creator to bootstrap the process - update the files (only talk about the main files) - update the user data flags
--- a/readme.rst	Wed Mar 27 10:37:34 2013 +0100
+++ b/readme.rst	Wed Mar 27 19:19:59 2013 +0100
@@ -2,174 +2,251 @@
-Automatically create variations of virtual machines depending
-on configuration files availables in repository :
+Automatically create variations of virtual machines images for a cloud
+infrastructure (OpensStack or Amazon are currently supported) using
+SaltStack_ based configuration files availables in a given Mercurial_
+repository, eg.:
-it also automatically gets the correct scripts from while doing so: 
+The AMI creator tool consist in a properly configured AMI to be
+instanciated with proper parameters passed as user data. This instance
+(which creation itself is described below) by default also updates
+itself from the base configuration repository (stored in the following
+Mercurial_ repository):
-Present state
+.. _SaltStack:
+.. _Mercurial:
-The first step is to edit/commit/push the file "variation.sls" so that a:
-salt-call state.sls variation.sls 
+How to create an AMI
-will create the machine you want.
+Creating an AMI is as simple as instanciating the ami-creator image in
+your cloud infrastructure with proper user-data content.
-You can also clone the repository and commit to another hg repo at the
-condition that this repository can be reached by http / https.
+What will be installed and configured in the built AMI image will
+depend on the configuration description that will be used as SaltStack
-Then, launch an instance of "ami-creation" on openstack. Use at
-least 5Gb of disk (add more if your sls file requires more).
+.. _highstate:
+So to summarize, the process of building a customized AMI image is:
-In the userdata, copy something like (if you do not put anything, what will be
-used will be the latest revision from the default repo):
+1. Edit/commit/push the file "variation.sls" and all its dependencies
+   in a Mercurial repository (must be accessible from the cloud
+   instance) so that a:
-====== START =======
+.. sourcecode:: bash
+     user@host:~$ salt-call state.sls variation.sls
-ami.hg_rev_variation: 249:9ed741a55f17
-ami.hg_address_variation: my_variation 
-ami.type: debian #or ubuntu
+   will create the machine you want.
-### optionnal ###
-keystone.user: usearname
-keystone.password: password
-keystone.tenant: tenant
-keystone.tenant_id: 00000000000000000000000000000000
-keystone.auth_url: ''
-keystone.insecure: False   #(optional)
+2. Then, launch an instance of the "ami-creator" image in your
+   cloud. Use at least 5Gb of disk (add more if your sls file requires
+   more).
+   In the userdata form of your instance, copy something like (if you
+   do not put anything, what will be used will be the latest revision
+   from the default repo)::
-### only if you know what you are doing ! ###
-====== END ========
-The kernel_id and initrd_id are used to specify the 
+     ====== START =======
+     ## salt-grains #interpreter flag
+     ### necessary unless you want the standard image ###
+     ami.hg_address_variation:
+     ami.hg_rev_variation: 9ed741a55f17 #optionnal
+ my_variation
+     ami.type: ubuntu #or debian
-you can add an ip to your instance, ssh to the machine and wait until
-everything in /mnt is unmounted. you can then upload what you want to glance
-with something like (don't forget to change the ids in the last tim or to
-include your credentials by downloading and running
+     ami.source_address:
+     ami.source_hash: md5=1d72ed9c56abb899be02e7cae0822f1d
-glance add is_public=false disk_format=aki container_format=aki \
-name=variation_kernel < kernel_variation
+     ### optionnal if your image uses existing kernel/ramdisk id in openstack ###
+     kernel_id:a5b900bd-c009-40b2-a763-32b32996b1ee
+     initrd_id:b953e40a-2605-402b-8663-c8556a5899f2
+     ### optionnal (ami_creation code update) ###
+     ami.hg_address_server:
-glance add is_public=false disk_format=ari container_format=ari \
-name=variation_ramdisk < ramdisk_variation
+     ### optionnal (necessary to upload to openstack) ###
+     keystone.user: usearname
+     keystone.password: password
+     keystone.tenant: tenant
+     keystone.tenant_id: 00000000000000000000000000000000
+     keystone.auth_url: ''
+     keystone.insecure: False   #(optional)
-glance add disk_format=ami container_format=ami name="variation-amd64" \
-kernel_id=278de93c-5276-410f-a221-667cc1443cf8 \
-ramdisk_id=b953e40a-2605-402b-8663-c8556a5899f2 < /mnt/variation.img
+     ====== END ========
+3. you can then connect to your instance and wait until everything in ``/mnt``
+   is unmounted. Upload to glance (openstack) is automatic if you provided the
+   correct user data. If you want to upload to amazon ec2, you need to follow
+   the following steps: 
-top.sls : top file for the ami_creator
+.. sourcecode:: bash
+     root@amibuilder:~#  
-ami_creator.sls : state file to have a correct ami-builder environment
+Main Files
+minion: must be copied to /etc/salt, provides a masterless configuration for salt
-ami_creator_pre.sls : state file applied before the creation of variation
+ami_pre: state folder (contains a few sls files) applied before the creation of variation
-ami_creator_post.sls : state file applied after the creation of variation for
+ami_creator.sls: state file to have a correct ami-builder environment
+ami_post: state folder applied after the creation of variation for
-variation.sls : variation file (should be modified depending on requirements)
- : top file for the chroot
+Additionnal Documentation
-Additionnal Documentation
+All the operations are currently done in /etc/rc.local. The content of the file
+is explained in the following lines.
-what is being done and where :
+The first step parses the user-data related to our ami
-in /etc/init.d/ec2-run-ami
+  /usr/bin/env python /etc/salt/srv/others/
-If the userdata is not a script and is not compressed content, Add the content
-of userdata as salt-grains (version and address from repo...)
+The next step updates the salt states, if necessary by getting the necessary
+files from the hg repository
-in /etc/rc.local
   salt-call state.highstate
   salt-call state.highstate
-(doing it twice is necessary to load new states)
-  salt-call vm_create.create_variation 
+(doing it twice is necessary to load new state/modules from the hg repository). Next,
-which contains : 
+  salt-call state.sls ami_pre
-  salt-call state.sls ami_creator_pre
-which mounts a debian image and creates all the necessary files inside. Then
-update the chroot:
+downloads and mounts an image and creates all the necessary files inside.
-  salt-call vm_create.update_image /mnt/variation
+  salt-call state.sls ami_creator
-and run your state file (if necessary, modify /etc/salt/srv/variation.sls which
-will be executed inside):
-  salt-call vm_create.variate_image /mnt/variation
+updates the chroot, install salt and run your state file. Finally
-which runs your variation inside the chroot. Finally run
+  salt-call state.sls ami_post
-  salt-call state.sls ami_creator_post
+extracts your image kernel and ramdisk and unmount and resize as necessary
-which will extract your image kernel and ramdisk and unmount and resize as necessary
   salt-call vm_create.upload to glance
-if the correct userdata is available, this sends the kernel, ramdisk and image
-to glance using the glance python client.
+if the correct userdata is available, it also uploads the kernel, ramdisk and image
+to glance using the glance python client. Uploading to amazon ec2 is not yet functionnal
+    1. add an amazon ec2 client to :
+        a. create and mount an EBS volume
+        b. copy the contente of the img file to this volume
+        c. unmount and detach the volume 
+        d. take a snapshot of the volume
+        e. register the snapshot as an ami in amazon
-The salt module from glance does not work in debian as ``glanceclient`` package
-is only available in experimental and I did not manage to connect to openstack
-essex with it.
+    2. include the bootstrap code as a variation sls file
+Bootstrapping an ami_creator machine
+The first step is to add the salt ppa to install it:
+.. sourcecode:: bash
+  root@machine~# add-apt-repository -y ppa:saltstack/salt
+The next step is to install salt-minion and mercurial:
-The salt module is working as intended.
+.. sourcecode:: bash
+  root@machine~# aptitude update && aptitude install salt-minion mercurial
+Then, clone the server repository:
+.. sourcecode:: bash
+  root@machine~# hg clone /etc/salt/srv
-Optimal behavior of this project
+Copy the necessary scripts to have launch the ami creation process at boot:
+.. sourcecode:: bash
+  root@machine~# cp /etc/salt/srv/others/rc.local /etc/rc.local
+and copy the configuration to make salt-minion masterless:
-3 levels of scripting:
+.. sourcecode:: bash
+  root@machine~# cp /etc/salt/srv/minion /etc/salt/
+save this machine as an ami-creator image and you can then use it by simply
+adding relevant user data at boot
+Manual upload to openstack (glance)
-Virtual Instance Launcher
+After the end of the creation process (nothing is mounted in /mnt/variation and
+you can see a .img, a kernel and a ramdisk in /mnt/output), you can upload your
+image to glance manually, provided you have credentials as a file
+downloaded from openstack.
+1. First, add your credentials to path (change ):
+.. sourcecode:: bash  
+  root@machine~# . /${correct_path}/ 
+2. Then, if necessary, define a name for your ami:
+.. sourcecode:: bash
+  root@machine~# export ami_name=variation
-(hooked to modifications of HG repository)
+3. And change to the correct directory:
+.. sourcecode:: bash
+  root@machine~# cd /mnt/output 
+4. If necessary, upload the kernel and the ramdisk:
-Ami Creator
+.. sourcecode:: bash
+  root@machine~# glance add is_public=false disk_format=aki container_format=aki \
+  name=${ami_name}_kernel < kernel_variation
-An image available on openstack / ec2.
+.. sourcecode:: bash
+  root@machine~# glance add is_public=false disk_format=ari container_format=ari \
+  name=${ami_name}_ramdisk < ramdisk_variation
-At boot, the user must supply a repository address in user data as
-``variation_address`` this information is used by salt at boot to download the
-correct revision of files.  it then execute salt-call
-vm_create.create_variation which will create a complete ami file and upload it
-to glance EC2
+5. Finally, upload the image after replacing the kernel_id and ramdisk_id
+   according to the previous uploads:
+.. sourcecode:: bash
+  root@machine~# glance add disk_format=ami container_format=ami name="${ami_name}" \
+  kernel_id=278de93c-5276-410f-a221-667cc1443cf8 \
+  ramdisk_id=b953e40a-2605-402b-8663-c8556a5899f2 < /mnt/variation.img
+Manual upload to ec2
+How to upload the ami you created on amazon manually (from_):
+1. Create an EBS volume of the minimal size to host the disk image
-Chroot Image
+2. attach the created volume to ami_creator: 
+3. transfer the content of the image to the EBS volume (replace xvdf by correct
+   device name):
-this script should only be a salt state file, and must be run using
-salt-highstate before packaging of the ami.
+.. sourcecode:: bash
+  ubuntu@machine:~$ sudo dd bs=16M if=/mnt/output/variation.img of=/dev/xvdf
+4. Detach the volume, take a snapshot, create an ami image from the amazon ec2
+   web interface. Make sure to use the correct "aki" (kernel image) as
+   indicated on documentation_ (for eu, use aki-71665e05).