Warning: This document describes an old release. Check here for the current version.

EC2 Client Notes

This section explains how to use 3rd-party EC2 and S3 clients with the Nimbus EC2 frontends, both SOAP and Query.

Uploading VM Images to Cumulus with Boto (#)

The python library boto can be used to upload images to Cumulus. Once a VM is staged into Cumulus it can be submitted for execution.

In order for Nimbus to efficiently find VM images in Cumulus a naming convention is used. Three pieces of information are needed to follow this convention:

  1. The repository bucket name
  2. By default this is Repo
  3. The image name prefix
  4. By default this is VMS
  5. Your canonical ID
  6. The site admin must provide this to you at the time of your account creation. It can also be found in your cloud clients configuration file (More Info)

That information is used to form a Cumulus url in the following way:

cumulus://<cumulus hostname>/<bucket name>/<prefix>/<canonical ID>/<image name>

Once you have the proper url formed you can use that information and boto to upload the image into the storage repository.

from boto.s3.key import Key
from boto.s3.connection import OrdinaryCallingFormat
from boto.s3.connection import S3Connection

s3id="<Cumulus ID>"
s3pw="<Cumulus password>"
host="<Cumulus hostname>"
port=<Cumulus port>
canonical_id="<Canonical ID>"

cf = OrdinaryCallingFormat()
s3conn = S3Connection(s3id, s3pw, host=host, port=port, is_secure=False, calling_format=cf)

bucket = s3conn.get_bucket("Repo")
k = Key(bucket)
k.key = "VMS/" + canonical_id + "/" + image_name

At this point your image will be ready for submission using the name image_name

Image Names

You may upload an image to any Cumulus name you wish and still submit it for run. This is described below. However, the above naming convention is strongly recommended in order to have a smooth and natural experience.

Uploading VM Images to Cumulus with s3cmd (#)

There is more information here.

Using the EC2 Query frontend from Python with Boto (#)

Nimbus 2.5 supports for the EC2 Query interface, which is used by the excellent Python client, boto. In order to use this interface, you must have Query credentials from the cloud administrator. These are composed of an access identifier (usually the hash of your DN) and a shared secret key.

You also need the URL of the Query interface on the Nimbus service node. See the University of Chicago cloud for example.

Before you can launch a VM you must first upload it to Cumulus. That process is described here

import boto
from boto.ec2.regioninfo import RegionInfo

# first create a region object and connection
region = RegionInfo(name="nimbus", endpoint="service.hostname.com")
conn =  boto.connect_ec2("YOUR_ACCESS_ID_HERE", "YOUR_SECRET_KEY_HERE",
                         port=SERVICE_PORT_NUMBER, region=region)

# then do something with the connection

Please refer to the boto documentation for more details on usage.

Image Names

Image names can be a full cumulus:// url, or simply the image name portion of the naming convention cumulus://<cumulus hostname>/<bucket name>/<prefix>/<canonical ID>/<image name>

A Full Example with Boto (#)

import boto
from boto.s3.key import Key
import time
from boto.s3.connection import OrdinaryCallingFormat
from boto.s3.connection import S3Connection
from boto.ec2.connection import EC2Connection
from boto.ec2.regioninfo import RegionInfo
access_id="<Cumulus ID>"
access_secret="<Cumulus password>"
host="<Cumulus hostname>"
port=<Service port: 8444>
cumulusport=<Cumulus port: 8888>
canonical_id="<Canonical ID>"

region = RegionInfo(name="nimbus", endpoint=host)
ec2conn = boto.connect_ec2(access_id, access_secret, region=region, port=port)
cf = OrdinaryCallingFormat()
s3conn = S3Connection(access_id, access_secret, host=host, port=cumulusport, is_secure=False, calling_format=cf)

# upload the file according to the Nimbus naming convention
bucket = s3conn.get_bucket("Repo")
k = boto.s3.key.Key(bucket)
k.key = "VMS/%s/%s" % (canonical_id, "my_image_name")
k.set_contents_from_filename(<path to image file>)

# now run the VM
reservation = ec2conn.run_instances("my_image_name")
instance = reservation.instances[0]

# poll for status
while instance.state != 'running':
    print 'instance is %s' % instance.state

# terminate the instance

Using the EC2 SOAP frontend from the console (#)

When using a cloud running the EC2 frontend, you can download this EC2 client from Amazon or try a number of different client that are out there.

EC2 Upgrades

Amazon EC2 upgrades happen without warning and so there is sometimes a sync error between their default tools and the tools needed to work with particular Nimbus elastic services.

With the client from Amazon, you need to adjust the following environment variables (using other clients, the configurations can vary).

  • EC2_HOME - Set the base directory (such that $EC2_HOME/bin is valid)
  • EC2_URL - Set the service endpoint to the Nimbus based service
  • EC2_CERT - Set the public credential to your public .pem based cert
  • EC2_PRIVATE_KEY - Set the private credential to an unencrypted .pem based key


$ export EC2_HOME=`pwd`
$ export EC2_URL=https://HOST:PORT/wsrf/services/ElasticNimbusService
$ export EC2_CERT=/tmp/ec2clientcert.pem
$ export EC2_PRIVATE_KEY=/tmp/ec2clientkey.pem

The URL for EC2_URL will be provided to you by an administrator or some documentation (HOST:PORT in the example is a placeholder for an actual hostname and port number)

For example, see the Nimbus cloud docs for running EC2 clients.

If you have a problem, consult the troubleshooting guide.

Before being able to launch an instance, you will need to register a keypair like so:

  • $ MYPUBKEY=`cat .ssh/id_rsa.pub`
  • Check to make sure those contents look OK:

    $ echo $MYPUBKEY
  • Send the public key value to the service, labelling it "mykey"

    $ ec2-add-keypair "mykey||$MYPUBKEY"

List the available images in your personal directory:

$ ec2-describe-images

Pick one, for example "hello-cloud". And with the "-k" argument, use the ssh key label you used above to register an ssh public key.

$ ec2-run-instances -k mykey hello-cloud

Check status:

$ ec2-describe-instances

After seeing 'pending' change to 'running', log in at the address printed:

$ ssh root@abc.def.com

Terminate using the "instance ID" printed when you first launched (and also available from ec2-describe-instances). It looks like "i-4662834e":

$ ec2-terminate-instances i-4662834e