Prepare Amazon AMI from Oracle VirtualBox vdi, Slackware 14 16-06-2014

Summary:

In my previous article, I tried to explain a step by step process for building up a Slackware Amazon AMI (loop device etc). One can read that article here . Just after that article, I had noticed that slackbuilds.org published the xen package for Slackware 14. I aimed to prepare same image for Slackware 14 with a different (probably a more clear cut) process. So, in this document, I tried to explain a way to prepare Slackware Amazon AMI from Oracle VirtualBox vdi file. What I have done, in short, install Slackware 14 on a virtualbox machine (say Guest 1), then attach this vdi file to an another VirtualBox machine(Guest2), and convert this vdi (for guest2 it is is not a file but a disk) to an image at Guest2 machine. After that, as I detailed the procedure at the previous article, I manipulate image and bundle, upload-bundle and register that image at Amazon AWS as an S3 backed AMI.
Before of ongoing steps,I want to express one important detail: At the end we have a S3 bucket (root device= instance store) image. If one can want to use that image , she/he must use small type instance as a smaller instance. But what if she/he need a micro instance instead of the small one.   This can be achieved by an ebs backed image and I will mention that at the end of the article.

Prerequisites:

Linux host machine, Oracle VirtualBox, Slackware 14 installation media,  Amazon AWS account, installed Amazon API and AMI tools.

Steps:

1. Installation of the VirtualBox machine
Install Slackware 14 (here 32 bit)  at the Guest 1. I don't get into details.  A 8Gb disk size seems to be enough. For Xen kernel, required (Or at least I assumed that they are required since I didnot tried one by one) groups of software are: a ap d f k l n t tcl and x. Grub has to be installed too. Xen and the other required packages can be found at slackbuild.org.
At the Guest2 , any linux distribution can be installed since all to tools used here can be found at almost all distributions. (In this work, I installed the Slackware again). This second machine's mission is to recognized the vdi as a physical disk and save it as an image file. For the same purpose, one can use host machine also by using VirtualBox's terminal commands (i.e. # VBoxManage internalcommands converttoraw ...)

2. Xen Installation
 Grup boot loader should be installed at the Guest 1  and be sure that it works (that is system can be opened via grub).  Get xen kernel package and all the other required packages from slackbuilds.org and install them on Guest 1.
Run the script which can be find under the xen/dom0 folder  (when installing Xen, you will see the folders).
Make sure that you can start the Xen system with the grub.
Finally, you can remove lilo and all the original kernel packages (not Xen) and all other packages which in fact not needed for  a server (especially under the sub group of x). Now we can close the Guest1 machine.

3. Prepare Amazon AMI (By using another VirtualBox machine: Guest2)
Note: Instead of using another guest machine one can do similarly steps under the Host machine also. In fact, that way has some advantages and makes the topic more clearer. That process are explained very well at the link http://www.niftiestsoftware.com/2011/09/24/converting-a-virtualbox-vm-to-an-ec2-ami/ .

Anyway, continuing on our way,
Lets add vdi file of the Guest1 machine to the Guest 2 machine as a disk (VirtualBox/Guest2/Settings/Storage/Controller/Harddisk). Please make sure that the newly added disk should be under the original Guest2 disk.  Otherwise, you may not open the Guest2 from its original disk.
At the Guest2 machine,
prepare working place:

# mkdir /tmp/amiworks
# mkdir /tmp/amiworks/fs-slackware
# cd /tmp/amiworks

prepare newly added disk  (here  in our exercise, that is /dev/sdb )

# fdisk /dev/sdb
    p
    w
# mount /dev/sdb1 /tmp/amiworks/fs-slackware

Modify /etc/fstab file as:

/dev/sda1 / ext3 defaults 1 1
none /dev/pts devpts gid=5,mode=620 0 0
none /dev/shm tmpfs defaults 0 0
none /proc proc defaults 0 0
none /sys sysfs defaults 0 0
/dev/sda2 /mnt ext3 defaults 0 0
/dev/sda3 swap swap defaults 0 0

Since we installed Slackware14 onGuest1 as default, our DHCP setting is already ok. (i.e.at the /tmp/amiworks/fs-slackware/etc/rc.d/rc.inet1 there is a line as USE_DHCP[0]="YES".)
We finished with the disk.

# umount /tmp/amiworks/fs-slackware/


Copy disk to image file.

# dd if=/dev/sdb1 of=/tmp/amiworks/slackami14-v7.img bs=16M

[Note: At this step, if you have a running ec2 instance then you can upload this image (probably with compressed version) to that ec2 instance and following ongoing steps at that machine. It not only saves a lot of time but also gives a chance to have ebs backed image which has some advantageous over s3 backed one]
Anyway, here we prefer to work with a local machine.

In order to create an AMI you need amazon ec2 ami and api tools, an amazon account and the account number(like 111222333444 without hyphen), private key and certificate files.
The x509 certificate files can be created and downloaded easily from AWS management console (under the access credential menu).

# ec2-bundle-image -i /tmp/amiworks/slackami14-v7.img -r i386  -k pk-privatekeyfile.pem -c cert-certificatefile.pem -u 111111111111

Before upload the files to Amazon AWS, you need to create a bucket at S3 service. I created a bucked named slackware-images. Again access and secret keys can be obtained from access credential menu of AWS management console.  Then finally you can register the prepared image (FYI, the upload process gets enormous time)

# ec2-upload-bundle -b slackware-images -m /tmp/slackami14-v7.img.manifest.xml -a -s
# ec2-register slackware-images/slackami14-v7.img.manifest.xml -n slackware-14-v.7

We are done. Clear working directory.

# rm -r /tmp/amiworks

With these last steps, we created ami image, uploaded it to Amazon AWS and registered it as an AMI. Now one can use it as an instance

Note: When initiate a new instance at AWS/EC2 from this image choose the kernel named as aki-f5cbe081.

What if we choose to work with ec2 machine

Above at the step where we copying the vdi file to image file we are still at the local virtualbox machine (Guest 2). There I mentioned that we can do the same thing at the local machine. Nevertheless, when working at the local machines we need to upload the files to s3 bucked and that step gets a lot of time. To overcome this obstacle, we may use ec2 machine as a working place (even if you don't have one, to prepare and run a new ec2 machine is really easy).
in short we compress the image file at the local machine and upload this compressed file to the ec2 instance then,  bundle, upload to s3 and register the image as always.
Lets do them:
Open up the Guest1 machine and enter the below command on its terminal

# cat /dev/zero > /tmp/zerofill

There we get a warning which says there is no space at the disk. It is not a problem. Then remove that zero file. (this way we will have a more compressed file)

rm /tmp/zerofill

Just before of closing the machine, one needs to modify the /etc/fstab file as described above and makes sure  DHCP is active. And last, shutdown the machine.

At the host machine

# VBoxManage internalcommands converttoraw slackami14.vdi ec2-slackami14.img

Compress this image file and upload to ec2 machine.

# tar cjf ec2-slackami14.img.tar.bz2 ec2-slackami14.img
# scp -i <secretkey.pem> ec2-slackami14.img.tar.bz2 root@:<path_to_working_dir>

Change the working machine to ec2 instance. At there expand the compressed file, bundle-image, upload to s3 and register as always.

# tar xf slackami14.img.tar.bz2
# ec2-bundle-image -i ec2-slackami14.img -r i386 -k <pk-privatekeyfile.pem> -c <cert-certificatefile.pem> -u
# ec2-upload-bundle -b slackami -m ec2-slackami14.img.manifest.xml -a -s
# ec2-register slackami/ec2-slackami14.img.manifest.xml -n slackware-14.v7 -K <pk-privatekeyfile.pem> -C <cert-certificatefile.pem>

If we aimed to create a ebs backed image instead of s3 one, then follow below steps.

Instead of  S3 (incident store), create ebs backed store

In this case, we really need an EC2 incident. Prepared (optionally compressed) image will be upload to ec2 instance. And following the steps detailed at 
http://www.niftiestsoftware.com/2011/09/24/converting-a-virtualbox-vm-to-an-ec2-ami/  gives us a ebs backed Amazon image

Last Words

Note: When initiate a new instance at AWS/EC2 from the image choose the kernel named as aki-f5cbe081. I didn't try all but most of them (I believe) do not work with our image.

Hope this explanations will help you. All comments, corrections etc are welcome. My email address is soydaner{at]kahverengi[dot]com.

I will also make this image to public at AWS but please be aware that this image is not suited for production environment but for educational purposes only