Nimbus TP2.2 Admin Quickstart

The quickstart assumes the services node you are working on is Linux or some UNIX variant (such as OSX perhaps). The Xen VMM nodes are required to be Linux. Most testing is carried out on Linux currently.

There are two system accounts involved:

  • privileged account - Pick a privileged account to run the remote client facing service. A privileged account on the VMM nodes is necessary as well. This guide assumes the privileged account on the service node and VMM nodes have the same name but that is not strictly necessary.

    In this guide (especially in the command samples) we will refer to an account of this type named nimbus with terminal prompts like "nimbus $"

  • superuser account - The root account is necessary to install dependencies on the VMM nodes (Xen, ebtables, etc.) and also to install the Nimbus agent that lives on the VMM nodes (workspace control).

    In this guide (especially in the command samples) we will refer to an account of this type named root with terminal prompts like "root #"

The quickstart is broken up into these steps:


Part I: Install/verify dependencies

I.A. Service container (#) (#)

Currently Nimbus is deployed into a Globus 4.0.x Java container which is a system built around the Apache Axis engine.

To get you started with this very quickly, we provide something called the AutoContainer which is a complete Globus Java web services core environment. It comes with a setup program that configures everything you need to get a secure web services container running from scratch in about a minute.

This includes creating a new certificate authority, creating a new host certificate, creating a new user certificate(s), and adjusting relevant configuration files.

Other (more time consuming) approaches:

  • If you are an intermediate Globus user, you may want to check out this section of the reference manual for a more manual approach.
  • If you are a veteran Globus user, you will probably already have a Java web services environment set up with security somewhere and can skip this section altogether.

* Acquire the AutoContainer: (#)

On the downloads page is a recent version of the AutoContainer.

* Run the AutoContainer: (#)

In the following steps, make sure you've expanded the directory to the place you want to leave it (absolute paths will be used for some configurations).

nimbus $ tar xzf autocontainer-GT4.0.8-v1.0.tar.gz
nimbus $ cd autocontainer
nimbus $ ./bin/setup-container.sh

That's it. The program will instruct you on any other necessary steps.

You'll end up being directed to run a test command whose output will look something like this if everything is set up:

nimbus $ /somewhere/autocontainer/bin/test-container.sh
Got notification with value: 3
Counter has value: 3
Got notification with value: 13

I.B. Xen (#)

The workspace service manages Xen VMs (KVM on its way).

The workspace service itself does not need to run on a Xen node. You must pick VMM nodes to install Xen on, the workspace service will manage these. For the quickstart, pick just one node to install and test with. When all goes well, add more into the pool of VMMs the workspace service can send work to.

You need to install Xen and get a basic VM running with bridged networking.

Nimbus has been most heavily tested with the Xen 3.1 series although it is known to be working with 3.2 (this may require the "tap:aio" configuration in workspace-control's "worksp.conf" configuration file). There is no information yet about Nimbus on the new/upcoming Xen 3.3 series.

You can actually put off the whole Xen step until later if you just want to evaluate the Nimbus clients and services: right out of the box Nimbus makes "fake" calls to workspace control (which is the agent that actually manages things on the VMM nodes). This allows the service side to be tested independently of the VMM. You "join them up" by turning off the "fake" switch when you're ready to do end to end testing.

* Acquire and install Xen: (#)

Your Linux distribution might have good support for Xen (Debian/Ubuntu, RedHat, Gentoo, SUSE, etc.) in which case it is best to defer to distribution specific notes about how to get Xen up and running (with bridged networking).

If your Linux distribution doesn't support Xen well or you are having some inexplicable problem, you can always try installing from source or binary, starting at http://xen.org/

* Test Xen: (#)

You can test Xen with any Xen VM image, mostly. In this guide we will use a small and simple VM image as an example. Look here for some ideas about where to find Xen VM images. You can also create them from scratch of course, if you have the time.

In particular, we will be using a ttylinux-based image. ttylinux is a very small, yet functional, Linux distribution requiring only around 4 MB. Visit the ttylinux home page for a list of some of its many nice features.

You can download a tarball containg the image here. This directory should contain the following files:

  • ttylinux-xen: The VM image
  • ttylinux-xen.conf: A sample Xen configuration for the VM image

Take into account that the provided ttylinux image is not the exact same image you can download from the ttylinux home page. It is preconfigured to obtain a network address through DHCP, and depends on a different root device than a regular ttylinux image (to make the image Xen-friendlier).

Test the image using the provided Xen configuration file. First of all, make sure you replace the values of the kernel and disk parameters inside the file with appropriate values. In particular, you should point kernel to your Xen guest kernel and disk to the location of the ttylinux-xen file. The contents of the configuration file should look something like this:

kernel = "/boot/vmlinuz-2.6-xen"
memory = 64
name = "ttylinux"
disk = ['file:/root/ttylinux-xen,sda1,w']
root = "/dev/sda1 ro"
vif = ['']

To test the image, run the following:

root # xm create ttylinux-xen.conf -c

You should see ttylinux boot messages, followed by a login prompt. You can log in using user 'root' and password 'root'. You can use ifconfig to check that the networking interface (eth0) is correctly setup. If you do not have a DHCP server, you can also use ifconfig to configure eth0 with a static IP address. Once the network is correctly set up, you should be able to ping out to some other machine besides dom0 and from that other machine be able to ping this address. Note that you should do all this just for the purposes of verifying that the image works correctly.

Keep the image configured to obtain a network address automatically via DHCP (even if your network doesn't have a DHCP server), as the Workspace Service will use DHCP to dynamically set up networking in it.

Note again: most Xen documentation you will find online about testing and troubleshooting the install is valid in this situation because you are not doing anything specifically related to Nimbus here.

I.C. Misc. libraries for the VMM nodes (#)

There are a few libraries needed on the VMM nodes before workspace control will work.

The ISC DHCP server (or DHCP server with compatible conf file) and ebtables are required to be running on each hypervisor node.

Any recent version of each package should be compatible, the scripts distributed with workspace control that automate the configurations were tested with ISC DHCP 3.0.3 and ebtables 2.0.6 userspace tools.

For more information on why this software is now necessary and how it will not interfere with a site's pre-existing DHCP server, see the network configuration details section in the reference guide.

Since these two pieces of software are relatively common, they may already be present on your hypervisor nodes via the package management system. Check your distribution tools for packages called dhcp (ISC DHCP server) and ebtables. You can also check for the existence of /sbin/ebtables or /usr/sbin/ebtables (for ebtables) and any of the following files for the DHCP server:

  • /etc/dhcp/dhcpd.conf

  • /etc/dhcp3/dhcpd.conf

  • /etc/init.d/dhcpd

  • /etc/init.d/dhcp3-server

If these software packages are not installed, all major Linux distributions include them and you should be able to easily install them with your package management system. For example, "rpm -ihv dhcp-*.rpm", "apt-get install dhcp", "emerge dhcp", etc. And similarly for ebtables.

ebtables requires kernel support in dom0, the default Xen kernel includes this support. If your dom0 kernel does not include these for some reason, the options to enable are under Networking :: Networking options :: Network packet filtering :: Bridge Netfilter Configuration :: Ethernet Bridge Tables

workspace-control also requires sudo and Python 2.3+ on all VMM nodes under the control of the Workspace Service.

I.D. SSH (#)

Finally, the privileged account on the service node needs to be able to SSH freely to the VMM nodes (without needing a password). And vice versa, the privileged account on the VMM nodes needs to be able to freely SSH back to the service nodes to deliver notifications.

You will be given the opportunity to get this right later during the service auto-configuration steps. The relevant security setups will be tested interactively, making sure you get this right.


Part II: Install/verify the Nimbus service package (#)

II.A. Download and install (#)

* Retrieve and unpack:

Grab the main Nimbus archive from the downloads page and unpack the archive:

nimbus $ wget http://workspace.globus.org/downloads/nimbus-TP2.2.tar.gz
nimbus $ tar xzf nimbus-TP2.2.tar.gz

* Set the target container: (#)

Now navigate into the new directory and set the "GLOBUS_LOCATION" variable to the target container directory you installed above:

nimbus $ cd nimbus-TP2.2
nimbus $ export GLOBUS_LOCATION=/path/to/globus_location

If you installed the AutoContainer, this would look something like this:

nimbus $ export GLOBUS_LOCATION=/somewhere/autocontainer/gt/ws-core-4.0.8/

Alternatively you could get that same GLOBUS_LOCATION set in your environment by sourcing this file:

nimbus $ source /somewhere/autocontainer/bin/source-me.sh

(a quick examination of the source-me.sh script will reveal that there is not much to that)

* Build and install: (#)

There are several options in the "bin" directory (see the toplevel README file). Choosing "./bin/all-build-and-install.sh" is all you should need to do.

nimbus $ ./bin/all-build-and-install.sh

A log of a successful build can be seen here.

If you look into that output, you can see that several Nimbus components are built and installed by default:

  • The Java based RM API and workspace service - VM/VMM manager
  • Default clients
  • WSRF frontend - a remote protocol implementation compatible with the default clients
  • EC2 frontend - a remote protocol implementation compatible with EC2 clients

For more information about what these things are, see the FAQ.

You will not be able to run against the EC2 frontend until you have set up the cloud configuration because it relies on users having a personal repository directory (out of scope of this document).

II.B. Necessary configurations (#)

There are a few configurations that cannot have any defaults.

As of Nimbus TP2.2, we provide an auto-configuration program to gently take you through these configurations. During the process, several tests will be made to verify your set up.

This is installed by default, run the following command to get started:

nimbus $ $GLOBUS_LOCATION/share/nimbus-autoconfig/autoconfig.sh

Otherwise you can refer to this section of the reference guide to see the old instructions. Those instructions may shed some light on certain configurations as you move past the testing stage and want to know more about what is happening (but honestly, the best place to look for configuration insight is in the .conf file inline comments).

* Authorization: (#)

In the beginning of the quickstart you may have edited a grid-mapfile to authorize a client to make a secure call to the container? Or if you are using the AutoContainer, the container-wide grid-mapfile can be found like so:

nimbus $ grep gridmap $GLOBUS_LOCATION/etc/globus_wsrf_core/global_security_descriptor.xml

A grid-mapfile is basically an access control list. It says which remote identities can access the container or a specific service.

There is a grid-mapfile specific to Nimbus set up by default. You can make Nimbus use the master container grid-mapfile that is configured OR you can keep it as a specific policy for Nimbus only.

To make it the same as the master container policy, edit the following file to point at the pre-existing grid-mapfile:

nimbus $ nano -w $GLOBUS_LOCATION/etc/nimbus/factory-security-config.xml

To keep it unique to Nimbus, edit the following grid-mapfile and add identities.

nimbus $ nano -w $GLOBUS_LOCATION/etc/nimbus/nimbus-grid-mapfile

If you used the AutoContainer, you can just copy the the grid-mapfile we found above (that grep command) onto this path, replacing the default "nimbus-grid-mapfile"

In the cloud configuration you will see that there is a handy way to make the grid-mapfile just a basic entry barrier. The real authorization decision can be made by more fine grained policies on a per user basis (for example, limiting certain users to certain amounts of total VM time, etc.).

II.C. Test call (#)

If you used the auto-configuration program, you are not ready for a live test yet. You will come back to this section after configuring workspace-control. Jump there now.

If you did not follow the auto-configuration program, currently Nimbus is set to "fake" mode. This allows you to get the service and VMM nodes working independently before you "join them up" for the live end to end test.

* New services: (#)

Starting the container again will result in these new services:

https://10.20.0.1:8443/wsrf/services/ElasticNimbusService
https://10.20.0.1:8443/wsrf/services/WorkspaceContextBroker
https://10.20.0.1:8443/wsrf/services/WorkspaceEnsembleService
https://10.20.0.1:8443/wsrf/services/WorkspaceFactoryService
https://10.20.0.1:8443/wsrf/services/WorkspaceGroupService
https://10.20.0.1:8443/wsrf/services/WorkspaceService
https://10.20.0.1:8443/wsrf/services/WorkspaceStatusService

* Client: (#)

In another terminal, set up the environment variables for GLOBUS_LOCATION and X509_CERT_DIR (as needed) and run this program:

nimbus $ cd $GLOBUS_LOCATION
nimbus $ ./bin/workspace
Problem: You must supply an action.
See help (-h).

OK, let's check out help, then.

nimbus $ ./bin/workspace -h

See sample output here.

A lot of options. This is the scriptable reference client. The cloud client is more user friendly. The cloud configuration which supports the cloud client is the recommended setup to provide users access to. This is very much because the cloud client offers a low entry barrier to people that just want to start getting work done.

For this quickstart we are going to run two of the actions listed in help, --deploy and --destroy. You can experiment with other ones, each action has its own help section.

(#)

* Deploy test: (#)

Grab this test script.

nimbus $ wget http://workspace.globus.org/vm/TP2.2/admin/test-create.sh

Run it:

nimbus $ sh test-create.sh

Sample successful output is here

As you can see, an IP address was allocated to the VM, a schedule given, and then some state changes reported.

If you opened a third terminal to destroy, you could see the state change move after the destruction, too. Or you could type CTRL-C to exit this command and run destroy in the same terminal.

nimbus $ $GLOBUS_LOCATION/bin/workspace -e test.epr --destroy

... and we get something like:

Destroying workspace 2 @ "https://10.20.0.1:8443/wsrf/services/WorkspaceService"... destroyed.

If you look at this file, you will now see some usage recorded:

nimbus $ cat $GLOBUS_LOCATION/var/nimbus/accounting-events.txt

The "CREATED" line is a record of the deployment launch. A reservation for time is made.

The "REMOVED" line is a record of the destruction. A recording of the actual time used is made. These actual usage recordings stay long term in an internal accounting database and (along with any current reservations) can be used to make authorization decisions on a per-user basis.


Part III: Install/verify workspace-control (#)

III.A. Download and install (#)

Download the "Control Agents" tar file from the download page, and untar it. The workspace-control directory contains multiple files, but we will be interested in only two of them: the install.py script which we will use to install workspace-control, and the worksp.conf.example file, a sample configuration file which we will need to edit to match the Xen installation.

Quickstart assumes you are using "/opt/workspace" as the target directory of the install. You need to be root in order to run the installer.

* Choose active config file: (#)

First copy over the sample config file:

root # cd workspace-control
root # mkdir /opt/workspace
root # cp worksp.conf.example /opt/workspace/worksp.conf

* Create privileged user and unix group: (#)

Next, you need to choose (or create) a user and unix group that will be used to run the backend script (for this guide, let's assume that your user and group are both called nimbus). Since we do not allow the backend script to be run as root, this user will rely on sudo to run Xen commands and other privileged commands. The installer will check all related permissions for you, create the necessary directories in /opt/workspace, and then adjust their permissions to be correct. In particular, the backend script (workspace-control) and other tools will be placed in directory /opt/workspace/bin, which will be owned by root. For more details on permissions, you can find a detailed note on this topic in the configuration file itself.

* Run install.py: (#)

Now we are ready to run the installer. The install.py program has three modes of operation you can use to install the backend script:

    -n, --noninteractive  Don't ask the user anything (for automated install with
                          a well-known conf file).

    -o, --onlyverify      Just run the setup tests and print what would have
                          happened in --noninteractive mode.  Good option to try
                          first since it will do nothing to the filesystem.

    -i, --install         Install the program, make needed directories, and set
                          them with default permissions. Will block to ask you
                          questions if necessary.

The --noninteractive mode is a good, fast choice if you want to use the default /opt/workspace hierarcy that comes in the sample configuration file. For example, you could run the installer like this:

root # python install.py -c /opt/workspace/worksp.conf -a nimbus -g nimbus -n

The "-a" flag is the privileged account to enable. The "-g" flag is the priviliged group to enable. The "-c" flag is the pre-existing config file (at its final location).

Just before exiting, the installer will print out the sudo policies you need to configure (see the next section).

To see more installation options, you can run install.py with the --help argument.

III.B. Necessary configurations (#)

* Configure sudo:

Using the visudo command, add the sudo policies printed out by the installer to the /etc/sudoers file. These policies should look something like this:

nimbus ALL=(root) NOPASSWD: /opt/workspace/bin/dhcp-config.sh
nimbus ALL=(root) NOPASSWD: /opt/workspace/bin/mount-alter.sh
nimbus ALL=(root) NOPASSWD: /usr/sbin/xm
nimbus ALL=(root) NOPASSWD: /usr/sbin/xend

These policies reflect the user that will be running workspace control (nimbus) and the correct full paths to the dhcp-config.sh and mount-alter.sh tools, the xm program, and the xend daemon (needed only in the case where the workspace-control program is permitted to reboot the daemon if it has fallen).

You may need to comment out any "requiretty" setting in the sudoers policy:

#Defaults    requiretty

The commands run via sudo are not using a terminal and so if you have "requiretty" enabled, this can cause a failure.

* Configure DHCP: (#)

DHCP is used here as a delivery mechanism only, these DHCP servers do NOT pick the addresses to use on their own. Their policy files are dynamically altered by workspace-control as needed. Policy additions include the MAC addresses which is used to make sure the requester receives the intended DHCP lease.

Configuring the DHCP server consists of copying the example DHCP file "dhcp.conf.example" (included in the workspace-control directory) to "/etc/dhcp/dhcpd.conf" and editing it to include the proper subnet lines (see the contents of the example file). The subnet lines are necessary to get the DHCP server to listen on the node's network interface. So, make sure that you add a subnet line that matches the subnet of the node's network interface. No lease configurations, available ranges, etc. should be added: these are added dynamically to the file after the token at the bottom.

In most cases it is unecessary, but if you have a non-standard DHCP configuration you may need to look at the "dhcp-config.sh" script in the protected workspace bin directory and look at the "adjust as necessary" section. The assumptions made are as follows:

  • DHCP policy file to adjust: "/etc/dhcp/dhcpd.conf"
  • Stop DHCP server: "/etc/init.d/dhcpd stop"
  • Start DHCP server: "/etc/init.d/dhcpd start"
  • The standard unix utility "dirname" is assumed to be installed. This is used to find the workspace-control utilities "dhcp-conf-alter.py" and "ebtables-config.sh", we assume they are in the same directory as "dhcp-config.sh" itself. Paths to these can alternatively be hardcoded to fit your preferred configuration.

The "aux/foreign-subnet" script (in the workspace control source directory) may be needed for DHCP support. It allows VMMs to deliver IP information over DHCP to workspaces even if the VMM itself does not have a presence on the target IP's subnet. This is an advanced configuration, you should read through the script's leading comments and make sure to clear up any questions before using. It is particularly useful for hosting workspaces with public IPs where the VMMs themselves do not have public IPs. This is because it does not require a unique interface alias for each VMM (public IPs are often scarce resources).

* Configure kernel(s): (#)

Copy any kernels you wish to use to the /opt/workspace/images directory, and list them in the guestkernels option under the [images] section of the /opt/workspace/worksp.conf configuration file. By doing this, clients can choose from these kernels in the metadata, but they must already exist at the hypervisor node and must be in the guestkernels list.

* Configure network(s): (#)

In the workspace-control /opt/workspace/worksp.conf configuration file, find the association_ settings in the [networking] section. If xenbr0 is the 'public' interface, which is the usual default,add this line:

association_0: private; xenbr0; vif0.0 ; none; 192.168.0.0/24

This specifies that there will be an network called 'private' with bridge 'xenbr0' with MAC address prefixes of 'none' (which means allow the service to decide) given to virtual NICs bound to this bridge and with an authorized IP range of 192.168.0.0 to 192.168.0.255. This should match the service network settings.

The third field lists the interface of the NIC that is listening for DHCP requests. This is the interface that DHCP requests originating from VMs should be allowed to broadcast to. Directing the request to a specific interface prevents DHCP requests from being broadcast to other VMs as well as to the real network. You can doublecheck what vif is on what bridge by running "brctl show". Normally you'd run the DHCP server in dom0 and the defaults are like so:

xenbr0 - vif0.0
xenbr1 - vif0.1
xenbr2 - vif0.2

You are not required to run the DHCP server in dom0, the workspaces' DHCP request broadcast can be directed to any interface, but currently to run it in another domain would require altering the dhcp-config.sh callout to work remotely.

Note: - if you used the auto-configuration program (see above), then you may have been given a sample line for the "association_0" setting.

III.C. Testing (#)

For testing with a real VM (see testing section below) using that test-create.sh script, add your test VM from the Xen section.

The script is expecting a file named "ttylinux-xen" in the /opt/workspace/images directory because of the file://ttylinux-xen line in the xml definition file the script points to ($GLOBUS_LOCATION/share/nimbus-clients/sample-workspace.xml).


Part IV: End to end test (#)

Now revisit the service Test call section.

If you did not use the auto-configuration program:

You are now ready to turn "fake" mode off and try a real VM launch. Back on the Nimbus services node:

nimbus $ nano -w $GLOBUS_LOCATION/etc/nimbus/workspace-service/other/common.conf

... and change the "fake.mode" setting to "false"

fake.mode=false

Now revisit the service Test call section.





Do not hesitate to contact the workspace-user mailing list with problems.

We plan to streamline some of the steps and also significantly add to the troubleshooting and reference sections.