Warning: This document describes an old release. Check here for the current version.

EC2 Client Notes

This section explains how to use 3rd-party EC2 and S3 clients with the Nimbus EC2 frontends, both SOAP and Query.

• Uploading VM Images to Cumulus with Boto

• Uploading VM Images to Cumulus with s3cmd

• Using the EC2 Query frontend from Python with Boto

• A Full Example with Boto

• Using the EC2 SOAP frontend from the console

• Using spot instances

Uploading VM Images to Cumulus with Boto (#)

The python library boto can be used to upload images to Cumulus. Once a VM is staged into Cumulus it can be submitted for execution.

In order for Nimbus to efficiently find VM images in Cumulus a naming convention is used. Three pieces of information are needed to follow this convention:

1. The repository bucket name
By default this is Repo
2. The image name prefix
By default this is VMS
3. Your canonical ID
The site admin must provide this to you at the time of your account creation. It can also be found in your cloud clients configuration file (More Info)

That information is used to form a Cumulus url in the following way:

cumulus://<cumulus hostname>/<bucket name>/<prefix>/<canonical ID>/<image name>

Once you have the proper url formed you can use that information and boto to upload the image into the storage repository.

from boto.s3.key import Key
from boto.s3.connection import OrdinaryCallingFormat
from boto.s3.connection import S3Connection

s3id="<Cumulus ID>"
s3pw="<Cumulus password>"
host="<Cumulus hostname>"
port=<Cumulus port>
canonical_id="<Canonical ID>"

cf = OrdinaryCallingFormat()
s3conn = S3Connection(s3id, s3pw, host=host, port=port, is_secure=False, calling_format=cf)

bucket = s3conn.get_bucket("Repo")
k = Key(bucket)
k.key = "VMS/" + canonical_id + "/" + image_name
k.set_contents_from_filename(path_to_image)

At this point your image will be ready for submission using the name image_name

Uploading VM Images to Cumulus with s3cmd (#)

There is more information here.

Using the EC2 Query frontend from Python with Boto (#)

Nimbus supports the EC2 Query interface which is used by the excellent Python client, boto. In order to use this interface, you must have Query credentials from the cloud administrator. These are composed of an access identifier (usually the hash of your DN) and a shared secret key.

You also need the URL of the Query interface on the Nimbus service node. See the University of Chicago cloud for example.

Before you can launch a VM you must first upload it to Cumulus. That process is described here

import boto
from boto.ec2.regioninfo import RegionInfo

# first create a region object and connection
region = RegionInfo(name="nimbus", endpoint="service.hostname.com")
conn =  boto.connect_ec2("YOUR_ACCESS_ID_HERE", "YOUR_SECRET_KEY_HERE",
                         port=SERVICE_PORT_NUMBER, region=region)

# then do something with the connection
conn.run_instances("image_name")

Please refer to the boto documentation for more details on usage.

A Full Example with Boto (#)

import boto
from boto.s3.key import Key
import time
from boto.s3.connection import OrdinaryCallingFormat
from boto.s3.connection import S3Connection
from boto.ec2.connection import EC2Connection
from boto.ec2.regioninfo import RegionInfo
    
access_id="<Cumulus ID>"
access_secret="<Cumulus password>"
host="<Cumulus hostname>"
port=<Service port: 8444>
cumulusport=<Cumulus port: 8888>
canonical_id="<Canonical ID>"

region = RegionInfo(name="nimbus", endpoint=host)
ec2conn = boto.connect_ec2(access_id, access_secret, region=region, port=port)
cf = OrdinaryCallingFormat()
s3conn = S3Connection(access_id, access_secret, host=host, port=cumulusport, is_secure=False, calling_format=cf)

# upload the file according to the Nimbus naming convention
bucket = s3conn.get_bucket("Repo")
k = boto.s3.key.Key(bucket)
k.key = "VMS/%s/%s" % (canonical_id, "my_image_name")
k.set_contents_from_filename(<path to image file>)

# now run the VM
reservation = ec2conn.run_instances("my_image_name")
instance = reservation.instances[0]

# poll for status
while instance.state != 'running':
    print 'instance is %s' % instance.state
    time.sleep(30)
    instance.update()

# terminate the instance
reservation.instances[0].terminate()

Using the EC2 SOAP frontend from the console (#)

When using a cloud running the EC2 frontend, you can download this EC2 client from Amazon or try a number of different client that are out there.

• Nimbus 2.7+ supports the "2010-08-31" WSDL (this EC2 client).
• Nimbus 2.3 - 2.6 supports the "2009-08-15" WSDL (this EC2 client) .
• Nimbus TP2.2 supports the "2008-05-05" WSDL (this EC2 client).
• The older Nimbus TP2.0 supports the "2008-02-01" WSDL (this EC2 client).
• So the default client may not always be the one to use. See a specific cloud's documentation for the definitive tools URL (for example, the Nimbus cloud). And see enhancement 6558.

With the client from Amazon, you need to adjust the following environment variables (using other clients, the configurations can vary).

• EC2_HOME - Set the base directory (such that $EC2_HOME/bin is valid)
• EC2_URL - Set the service endpoint to the Nimbus based service
• EC2_CERT - Set the public credential to your public .pem based cert
• EC2_PRIVATE_KEY - Set the private credential to an unencrypted .pem based key

Example:

The URL for EC2_URL will be provided to you by an administrator or some documentation (HOST:PORT in the example is a placeholder for an actual hostname and port number)

For example, see the Nimbus cloud docs for running EC2 clients.

If you have a problem, consult the troubleshooting guide.

Before being able to launch an instance, you will need to register a keypair like so:

• $ MYPUBKEY=`cat .ssh/id_rsa.pub`

• Check to make sure those contents look OK:

  $ echo $MYPUBKEY

• Send the public key value to the service, labelling it "mykey"

  $ ec2-add-keypair "mykey||$MYPUBKEY"

List the available images in your personal directory:

Pick one, for example "hello-cloud". And with the "-k" argument, use the ssh key label you used above to register an ssh public key.

Check status:

After seeing 'pending' change to 'running', log in at the address printed:

Terminate using the "instance ID" printed when you first launched (and also available from ec2-describe-instances). It looks like "i-4662834e":

Using spot instances (#)

Amazon EC2 spot instances (SIs) introduces a new approach to purchase virtual machine instances on Infrastructure as a Service clouds. In this approach, the cloud provider reserves part of the unused capacity for auction, and clients bid on this capacity. Each available VM type has an associated Spot Price, that changes periodically based on supply and demand. Clients with bids that exceed current SP gain access to available VMs for as long as their bid exceeds that price. More information on Amazon EC2 spot instances can be found at https://aws.amazon.com/ec2/spot-instances/.

You may also be interested in the spot instances introductory material in the Nimbus administrator's guide.

Nimbus introduces an open source implementation of spot instances that will allow science cloud users to save allocation credits in non-critical tasks during low-demand periods. Currently, the spot instances features are available in Nimbus through the following interfaces: Amazon EC2 WSDLs and Amazon EC2 Query API. Moreover, developers may want to link their own applications with Nimbus' spot instances through the RM API.

The Nimbus' spot instances operations are compliant with the WSDL schema version 2010-06-15 and higher. You can follow the instructions above in order to connect your preferred EC2 client to a Nimbus cloud.

Before submitting a spot instance request you should define a maximum bid for that request. There is no exact formula for defining an ideal bid. It basically depends on your needs and how much are you willing to pay to run your tasks.

The "price" is represented in Nimbus as a discount applied to minutes charged to your account. Thus, you likely do not want to bid above "1.0" since regular requests will always trump spot instances but your spot instance requests will be more "expensive" than the regular ones (this is akin to bidding above the normal instance prices on EC2).

The spot price history is a valuable information in determining the maximum bid for your requests. A good way to start is to place your bid between the historical maximum and minimum spot prices. If you want your instances to have a good chance of being executed straight away, you should place your bid somewhere near the maximum historical spot price. In case your tasks are not so urgent and can wait a couple of days (or weeks) to get executed on a low cost, you should place your bid somewhere close to the minimum historical spot price.

The Spot Price history is available in the EC2 APIs through the Describe Spot Price History operation. The supported parameters of this operation are:

• Start time: Start date and time of the spot instance price history data. (optional). Date string should conform with the ISO 8601 standard.
• End time: End date and time of the spot instance price history data. (optional). Date string should conform with the ISO 8601 standard.
• Instance type: The instance type of the spot instance price history data (optional). Currently Nimbus only supports one type of spot instance per site. Please contact your administrator to know which spot instance type is supported in your site.

SOAP Client (Amazon)

Operation command and parameters:

This example creates a spot instances request for ten m1.small instances.

$ ec2-request-spot-instances --price 0.50 myimage.img --key MyKeypair --instance-type m1.small --instance-count 10 --type one-time
SPOTINSTANCEREQUEST sir-f102a405 0.50 one-time active 2009-12-12T22:58:47+0200 i-3597b470 myimage.img m1.small default

Query Client (Boto)

Operation signature and parameters:

Example:

import boto
from boto.ec2.regioninfo import RegionInfo

# first create a region object and connection
region = RegionInfo(name="nimbus", endpoint="service.hostname.com")
conn = boto.connect_ec2("YOUR_ACCESS_ID_HERE", "YOUR_SECRET_KEY_HERE", port=SERVICE_PORT_NUMBER, region=region)

# Creates a spot instances request for ten m1.small instances.
request = conn.request_spot_instances(0.50, 'myimage.img', key_name='MyKeypair', instance_type='m1.small')

Retrieving spot instance Requests

Once submitted, you can check the status of your spot instance requests through the Describe spot instance Requests operation. Besides request launch information, this operation returns the request state (open, active, canceled or failed) and the id of the spot instance (in case the request is in the active state). Executing this operation without parameters will return information about all spot instance requests submitted by the user. Another way of using this operation is to request information about a single request or set of requests.

The supported parameter for this operation is:

• spot instance Request ID: Specifies the ID of the spot instance request to be described. (optional). This parameter can be a collection of spot instance request ids.

SOAP Client (Amazon)

Operation command and parameters:

This example returns information about current spot instance requests.

$ ec2-describe-spot-instance-requests sir-f102a405
SPOTINSTANCEREQUEST sir-f102a405 0.1 one-time active
2009-12-12T22:58:47+0200 i-3597b470 ami-7d3b6a38 m1.small default

Query Client (Boto)

Operation signature and parameters:

import boto
from boto.ec2.regioninfo import RegionInfo

# first create a region object and connection
region = RegionInfo(name="nimbus", endpoint="service.hostname.com")
conn = boto.connect_ec2("YOUR_ACCESS_ID_HERE", "YOUR_SECRET_KEY_HERE", port=SERVICE_PORT_NUMBER, region=region)

# Create a spot instances request for ten m1.small instances.
conn.request_spot_instances(0.50, 'myimage.img', count=10, key_name='MyKeypair', instance_type='m1.small')

# Describe spot instance requests, and print id and state
si_reqs = conn.get_all_spot_instance_requests()
print 'Nr. of Requests: ' + str(len(si_reqs))
print 'Id: ' + siReqs[0].id + ' State: ' + si_reqs[0].state

Output:

Nr. of Requests: 1
Id: sir-f102a405 State: active

Canceling Spot Instance Requests

The Cancel spot instance Requests operation cancels spot instance requests in the "active" or "open" state. Canceled requests aren't considered for fulfillment. However, running instances from a canceled request, will remain running until they are explicitly terminated or the spot price rises above the request bid. This operation can cancel a single request, or a set of requests.

The supported parameter for this operation is:

• spot instance Request ID: Specifies the ID of the spot instance request to be canceled. This parameter can be a collection of spot instance request ids.

SOAP Client (Amazon)

Operation command and parameters:

Example:

$ ec2-cancel-spot-instance-requests sir-ed3a2006
SPOTINSTANCEREQUEST sir-ed3a2006 canceled

Query Client (Boto)

Operation signature and parameters:

Example:

import boto
from boto.ec2.regioninfo import RegionInfo

# first create a region object and connection
region = RegionInfo(name="nimbus", endpoint="service.hostname.com")
conn = boto.connect_ec2("YOUR_ACCESS_ID_HERE", "YOUR_SECRET_KEY_HERE", port=SERVICE_PORT_NUMBER, region=region)

# Creates a spot instances request for ten m1.small instances.
spot_req = conn.request_spot_instances(0.50, 'myimage.img', count=10, type='one-time', key_name='MyKeypair', instance_type='m1.small')

# Cancel the submitted spot request
si_reqs = conn.cancel_spot_instance_requests([spot_req[0].id])

Managing Running Spot Instances

Spot instances perform exactly like ordinary Nimbus instances while running, and like other Nimbus instances, running spot instances can be managed (described or terminated) by the Nimbus Cloud Client and by usual EC2 API management operations, such as:

• ec2-describe-instances - Report on currently running instances.
• ec2-terminate-instances - Destroy currently running instances.

In the "Describe Instances" operation, spot instances will differ from ordinary running instances in the following attributes:

• Instance Life Cycle: 'spot' if the running instance is a spot instance, null or 'normal' otherwise.
• spot instance Request ID: the ID of the spot instance request, if the running instance is a spot instance, null otherwise.