Install VMM Software

On this page, you will install the Nimbus VMM software workspace-control and any of its dependencies (including Xen or KVM as well as libvirt, if these are not present already).

workspace-control contains scripts that will help you test that your Xen/KVM/libvirt installation will work with the programmatic access patterns that Nimbus uses.

You should use one VMM node for this exercise. When things are working end to end, you can propagate the configuration and dependencies to all of the other nodes on your cluster (assuming that they are roughly homogeneous).

Dependencies (#)

Before automating a VMM node with Nimbus software, it needs to be a VMM node. So the first thing we need to do is make sure that all the virtualization and networking dependencies are present.

The workspace service currently manages Xen or KVM VMs. It interfaces with both of these systems using a library called libvirt.

This guide will explain how to install the Nimbus workspace control software on top of both Xen 3.x and KVM. The instructions are intermingled so it is recommended that you first read our notes page for the hyper-visor that you intend to use:

There are many steps in this section which are common to both KVM and Xen. However there are also some slight differences mixed in with each step. We will visually distinguish the two with the following styles.

This is information needed for Xen

This is information needed for KVM

The reader can then key in on the information specific to their cloud.

Basic Dependencies

Install Xen 3.x or KVM as described in their respective notes above.

Install ebtables using your Linux distribution's package management tools.

ebtables in the host kernel

ebtables requires kernel support on the VMM machine. If your kernel does not include these for some reason one place to find the options to enable are under Networking :: Networking options :: Network packet filtering :: Bridge Netfilter Configuration :: Ethernet Bridge Tables

Install libvirt version 0.7 or higher using your Linux distribution's package management tools and user guides.

Older libvirt versions

There are differences in lower libvirt versions that have caused problems for some Nimbus users. If you take a chance with this and see an error about "createXML", you've probably got a very old version.

Python bindings for libvirt

An important step is to make sure that your libvirt installation supports the libvirt "python bindings". This is how the Nimbus software actually interacts with libvirt.

In some cases, the bindings (which come with a normal libvirt installation) are not installed. On some systems you need to trigger this to be there by installing the "python-dev" package before installing libvirt. In order to check you have the right install, Nimbus includes a script in the workspace-control download which you will run later (discussed below).

Make sure you have Python 2.4 or higher (but not Python 3.x).

If your Linux distribution doesn't support these packages well, you can try installing from source or binary, starting at libvirt.org (and either xen.org or linux-kvm.org) and ebtables.

Our next step in this guide will be to create a unix account for workspace-control and then install it. This will give us access to some scripts we can use to further verify the system is ready to be automated with Nimbus.

Bridged networking

You need to get a basic VM running with bridged networking. Using NAT networking for guest VMs will not be sufficient.

Choose/Create Privileged User Account (#)

There are two system accounts needed on the VMMs.

  • privileged account - Pick an account to make privileged (using the sudo program).

    In this guide (especially in the command samples) we will refer to an account of this type named nimbus with terminal prompts like "nimbus $"

    Note: this is not a privileged account in the sense that root is a privileged account. This is a regular account that you will give powers to during the setup process. It is an account that will be privileged because it will have control over the hyper-visor.

  • superuser account - The root account is necessary to install dependencies on the VMM nodes (Xen/KVM, ebtables, etc.) and also to install the Nimbus agent that lives on the VMM nodes (workspace control).

    In this guide (especially in the command samples) we will refer to an account of this type named root with terminal prompts like "root #"

    Note that root is not needed on the service node, just here on the VMM.

Set up libvirt permissions (#)

OK, you've got your non-root user nimbus created, now we need to allow it to control VMs using libvirt.

First, try out a libvirt command as root (it will not change anything if it succeeds or fails).

root # virsh -c 'xen+unix:///' list
root # virsh -c 'qemu:///system' list
Make sure this works as root before we try to enable the nimbus user to do this.

If that does not work, you will need to enable Xen to work with domain sockets. In the /etc/xen/xend-config.sxp file, activate the xend-unix-server configuration by including yes. Then uncomment the xend-unix-path setting and note the value (for example /var/lib/xend/xend-socket). Note that using Xen with the HTTP server method instead of this domain socket method is typically not secure.

Once it works as root, try as the nimbus user:

nimbus $ virsh -c 'xen+unix:///' list
nimbus $ virsh -c 'qemu:///system' list

You may hit an error about failing to connect to the hypervisor.

error: unable to connect to '/var/run/libvirt/libvirt-sock': Permission denied
error: failed to connect to the hypervisor

Why? Because you are not root. Check out the permissions on /var/run/libvirt/libvirt-sock

root # ls -l /var/run/libvirt/libvirt-sock
srwx------ 1 root root 0 Jul  1 21:28 /var/run/libvirt/libvirt-sock

The solution is to set the domain socket with certain permissions.

Read-only libvirt gotcha

If this works for you, make sure your setup is not just letting you list VMs right now, that will happen if the permissions are read-only for group or world.

You need to make sure it is both readable and writable for the root group and that the nimbus user is part of that group.

Note that we will test a full VM creation later which would uncover this difference.

If you are using later versions of libvirt you may see "Permission denied" errors. Be sure that in the libvirt qemu.conf file (usually /etc/libvirt/qemu.conf) be user you have the following settings:

user = "root"
group = "root"
dynamic_ownership = 0
clear_emulator_capabilities = 0

or (more secure):

user = "nimbus"
group = "nimbus"
dynamic_ownership = 0

First, make the nimbus user a member of the libvirtd group:

root # usermod -a -G libvirtd nimbus

Make the libvirt domain socket available for reading and writing by the libvirtd group by enabling the following configurations in /etc/libvirt/libvirtd.conf

unix_sock_group = "libvirtd"
unix_sock_ro_perms = "0770"
unix_sock_rw_perms = "0770"
unix_sock_dir = "/var/run/libvirt"

On some distributions you may need to specify auth_unix_rw = "none" if libvirt defaults to using PolicyKit.

libvirtd needs to be restarted for any changes to take effect. Validate your changes by checking the permissions with "ls" again.

Make sure the group change has taken effect with the nimbus user (using the "groups" command within the shell, you may need to relogin).

Try again:

nimbus $ virsh -c 'xen+unix:///' list
nimbus $ virsh -c 'qemu:///system' list

If you can still not do this basic list command as the nimbus user, look into your distribution's libvirt configurations and also don't hesitate to contact the workspace-user mailing list with problems.

Install workspace-control (#)

We can now download the VMM control package to the machine.

Download the "Control Agents" tar file from the download page, and untar it. This archive contains both workspace-control and the workspace pilot. For this configuration we are using workspace-control. You need to copy it to the /opt/nimbus destination directory like so:

root # curl -O http://www.nimbusproject.org/downloads/nimbus-iaas-controls-2.10.1.tar.gz
root # tar xfz nimbus-iaas-controls-2.10.1.tar.gz
root # mv nimbus-iaas-controls-2.10.1/workspace-control/* /opt/nimbus

Make sure the directory hierarchy is as the default configurations -- and this guide -- expect.

root # [ -f /opt/nimbus/bin/workspace-control.sh ] && echo "Correct."
Correct.

Now make sure the permissions are set up for running via sudo safely. Note the chown -R nimbus part, this is crucial for letting workspace-control do a lot of its work as the non-root user nimbus (only calling out to root via sudo to perform specific priviliged tasks).

root # cd /opt/nimbus
root # chown -R root bin etc lib libexec src
root # chown -R nimbus var
root # find . -type d -exec chmod 775 {} \;
root # find . -type f -exec chmod 664 {} \;
root # find bin sbin libexec -iname "*sh" -exec chmod 755 {} \;

Test the basic dependencies

Before moving on with a VM launching test, first we should confirm the basic assumptions about the environment. The workspace-control package contains a script that will examine the system.

Run the following command:

nimbus $ ./sbin/test-dependencies.sh

If this script reports you have a problem, look into your distribution's libvirt support (especially the python bindings that were discussed earlier) and don't hesitate to contact the workspace-user mailing list with problems.

Pick the kernel to use (#)

Pick a kernel in /boot that was installed with Xen to use for guest VMs. Let's say for example you find /boot/vmlinuz-2.6.16.29-xen was installed along with Xen.

It may have a matching initrd file, for example /boot/initrd-2.6.16.29-xen

In this case you would run:

nimbus $ KERNELDIR=/opt/nimbus/var/workspace-control/kernels
nimbus $ cp /boot/vmlinuz-2.6.16.29-xen $KERNELDIR/default
nimbus $ cp /boot/initrd-2.6.16.29-xen $KERNELDIR/default-initrd
The files under /boot may only be readable by root so you may need to become root to access them.

Now you have created a kernel called default that workspace-control will launch paravirtualized VMs with. The default kernel and initrd to use for VMs is controlled by the authz_kernels configuration in /opt/nimbus/etc/workspace-control/kernels.conf (by default the authorized kernel is "default").

Obtain the sample image (#)

A sample image has been prepared for testing. This is a basic VM that the Nimbus developers are familiar with, meaning it will be easier to get support help using this VM.

nimbus $ cd /tmp
nimbus $ curl -O http://www.nimbusproject.org/downloads/nimbus-z2c.gz
nimbus $ gunzip nimbus-z2c.gz

It is a minimal Debian VM with very little installed. It runs an SSH server when it boots. If you have console access to it, you will be able to log in to it using the root account with password root. It also runs the Nimbus context agent on boot, so you can use it to test contextualization.

nimbus $ cd /tmp
nimbus $ curl -O http://www.nimbusproject.org/downloads/ubuntu10.10.gz
nimbus $ gunzip ubuntu10.10.gz

This is an ubuntu 10.10 base install image with an on-board kernel. It runs ssh when it boots. However, root password logins are disabled (both on console and SSH). This image does not run the context agent.

Configure libvirt hypervisor (#)

Workspace control must be configured to use the correct libvirt hypervisor. This is done in /opt/nimbus/etc/workspace-control/libvirt.conf. The default hypervisor is Xen. If you are using KVM you'll need to edit the vmm property in this file.

[libvirt]
vmm: kvm0

Configure default bridge (#)

You set up Xen/KVM to use bridged networking, now it is time to check what the actual name of the bridge will be that the guest VM network card uses to get traffic to the LAN.

Using the brctl tool, you will see something like this in the list of bridges on the VMM system.

root # brctl show
bridge name     bridge id               STP enabled     interfaces
xenbr0          8000.feffffffffff       no              peth0

The vif (virtual interface) for domain 0 is vif0.0 and it is bridged to xenbr0. The guest VMs will by default be bridged in the same way.

This exact situation is common and happens to be the default bridge configuration in workspace-control's /opt/nimbus/etc/workspace-control/networks.conf file.

Notice default: xenbr0 in the [defaultbridge] section of that configuration file. The value here needs to be the bridge that VM traffic will be bridged to. In this case, the xenbr0 bridge.

# brctl  show
bridge name     bridge id               STP enabled     interfaces
virbr0          8000.000000000000       yes

Notice virbr0. This value must match the value for default: virbr0 in the [defaultbridge] section of that configuration file /opt/nimbus/etc/workspace-control/networks.conf. In that same file you must also change the value xen-ebtables-config.sh to kvm-ebtables-config.sh.

Configure sudo and ebtables script (#)

Before we can do a live test, we need to configure workspace-control to use sudo privileged properly.

Using the visudo command, add the necessary sudo policies. These policies reflect need to use the user that will be running workspace-control ("nimbus") and the correct full paths to the libexec tools. See /opt/nimbus/etc/workspace-control/sudo.conf for all the details and sample rules.

Make sure the sudo rules work without a password as the nimbus user.

nimbus $ /usr/bin/sudo /opt/nimbus/libexec/workspace-control/xen-ebtables-config.sh
ERROR: requires at least 2 arguments, syntax: add|rem <vifname> [<macaddr> <ipaddr> [<dhcpif>]]

nimbus $ /usr/bin/sudo /opt/nimbus/libexec/workspace-control/kvm-ebtables-config.sh
ERROR: requires at least 2 arguments, syntax: add|rem <vifname> [<macaddr> <ipaddr> [<dhcpif>]]

Seeing "ERROR" is a good thing here. If you see a password prompt, type in the nimbus user's password which hopefully is a "first time use" warning from sudo. Then try again. The goal is to perform this command from the nimbus account without needing any password.

You may need to comment out any "requiretty" setting in the sudoers policy, like so:

#Defaults    requiretty

The commands run via sudo are not using a terminal and so if you have "requiretty" enabled, this can cause a failure.

Obtain the workspace-control network sample (#)

In a previous section of this guide, you configured DHCPd and the network addresses to give to guest VMs.

The central IaaS services container wrote out the dhcpd.entries file that will be used for the DHCPd server configuration. In the same directory a sample file for workspace-control was written out, we will use this to test a VM launch via libvirt with your DHCPd server running. The file you need to obtain is called control.netsample.txt

Transfer this file from the central node to anywhere on the VMM node: $NIMBUS_HOME/services/var/nimbus/control.netsample.txt

We will use it in the next step. The example location on the VMM will be /tmp/control.netsample.txt.

Test VM creation with a generated libvirt config (#)

Once you have obtained the control.netsample.txt file, have the kernel configured and have the default bridge configured, you are ready to try starting the test VM using a libvirt configuration file.

This is not just any libvirt configuration file, but one representing exactly the configuration that the workspace-control tool will use to launch VMs on your system automatically.

A script called libvirt-xml.sh lives in /opt/nimbus/sbin that will generate XML that we can use to launch VMs using the libvirt commandline tool called virsh

It takes a few arguments:

nimbus $ cd /opt/nimbus
nimbus $ ./sbin/libvirt-xml.sh -h
Usage: see help (-h).

Options:
  --version         show program's version number and exit
  -h, --help        show this help message and exit
  --image=PATH      VM image file to test
  --netsample=PATH  Net sample file obtained from service node
  --memory=INT      Amount of MB RAM to assign
  --mountpoint=id   Mountpoint like 'sda1' to mount image to

That image we downloaded is at /tmp/nimbus-z2c or /tmp/ubuntu10.10, that will be the value for the --image flag.

The --memory value will be something easily doable by your system, e.g. 256.

The --mountpoint value will be sda1 if you are using xen with the nimbus-z2c image and hda if you are using kvm with the ubuntu10.10 image.

That leaves the --netsample flag. The value for this will be the path to the control.netsample.txt file we just obtained in the last step.

Run the following:

nimbus $ ./sbin/libvirt-xml.sh  --image /tmp/nimbus-z2c --netsample \
/tmp/control.netsample.txt --memory 256 --mountpoint sda1
nimbus $ ./sbin/libvirt-xml.sh  --image /tmp/ubuntu10.10 --netsample \
/tmp/control.netsample.txt --memory 256 --mountpoint hda

On stdout, you will see an XML document printed.

Now run the following script to reset things:

nimbus $ ./sbin/destroy-control-test.sh

This may need to be repeated to solve any problems that are being mentioned. Anytime you run libvirt-xml.sh, you should pair it with destroy-control-test.sh before trying again.

The output on stderr that looks like "libvir: Test error : Domain not found" is normal. This is an artifact of the tools scanning for information.

Having a problem? If the problem is not obvious from the output, the error and debug logs can be found like so:

nimbus $ ./sbin/cat-latest-logfile.sh

If putting the culprit error into a search engine does not produce any obvious answers, don't hesitate to contact the workspace-user mailing list. Describe your situation and any errors you are seeing. Attaching this debug log and including version information (of the VMM and libvirt and your OS) can often be helpful and reduce the amount of emails that need to occur.

Once you have the XML getting printed, capture it to a file:

nimbus $ ./sbin/libvirt-xml.sh --image /tmp/nimbus-z2c --netsample \
/tmp/control.netsample.txt --memory 256 --mountpoint sda1 > /tmp/z2c.xml
nimbus $ ./sbin/libvirt-xml.sh --image /tmp/ubuntu10.10 --netsample \
/tmp/control.netsample.txt --memory 256 --mountpoint hda > /tmp/z2c.xml

Remember to keep destroying the control-test

Trying libvirt-xml.sh but seeing an error like "The persistence file already exists"? This means you need to run that script: ./sbin/destroy-control-test.sh

Make sure the /tmp/z2c.xml file is what we expect. It should start with lines like this:

<?xml version="1.0" encoding="UTF-8" standalone="no" ?>
<domain type='test'>
    <name>control-test</name>

OK! Now as the nimbus user run the virsh command to boot up the VM.

nimbus $ virsh -c 'xen+unix:///' create /tmp/z2c.xml
nimbus $ virsh -c 'qemu:///system' create /tmp/z2c.xml

Now the VM will boot. If it gets the correct network address from DHCP you should be able to ping it and even log in over SSH.

First find the IP address we are expecting it to get:

nimbus $ grep ip /tmp/control.netsample.txt | awk '{print $2}'

Then give it a ping:

nimbus $ ping -c 1 192.168.0.2

If this does not work, the next step will be to log in via the console and troubleshoot what happened. You can log in using console to the nimbus-z2c image using the root account with password root

nimbus $ virsh -c 'xen+unix:///' console control-test
nimbus $ virsh -c 'qemu:///system' console control-test

Console permissions

With the reference image for KVM console access will not work. In order to use this you will need to create your own image

Console permissions

You may need to be root to run a virsh console subcommand because the nimbus user may not have access to the /dev/pts device being used for console access.

Does it not get an IP address from DHCPd? Is DHCPd running?

Did it get some other IP address from DHCPd? Your DHCPd may be misconfigured, perhaps it is not giving specific MAC addresses specific IP addresses. Perhaps you did not load it with the current dhcpd.leases entries from Nimbus.

If you get an error like "There is a VM running with this handle but there is no record of it in the program" then you may have been starting with virsh but not through libvirt-xml.sh. The way to correct this is to run virsh destroy control-test directly.

Seeing an error from a component, for example in the libvirt logs? If putting the error into a search engine does not produce any obvious answers, don't hesitate to contact the workspace-user mailing list. Describe your situation and any errors you are seeing. Including version information (of the VMM and libvirt and your OS) can often be helpful and reduce the amount of emails that need to occur.

When all is well, you should be able to log in as root over SSH. Trying this from off the node is the best final test.

ssh root@192.168.0.2

Final test: VM creation using workspace-control (#)

The final test will be to launch the same file with the same parameters, but using workspace-control.sh just as the service will. We will use a wrapper program called control-test.sh to invoke workspace-control.sh.

If you have a control-test VM running at the moment, shut it down using the kill script:

./sbin/destroy-control-test.sh

Gather the same arguments you used for ./sbin/libvirt-xml.sh

$ ./sbin/control-test.sh --image /tmp/nimbus-z2c --netsample \
/tmp/control.netsample.txt --memory 256 --mountpoint sda1

$ echo $?
0
$ ./sbin/control-test.sh --image /tmp/ubuntu10.10 --netsample \
/tmp/control.netsample.txt --memory 256 --mountpoint hda

$ echo $?
0

If the exit code is zero, we are good to go. SSH again to test for sure. Run the destruction script and you can move along to the next section in this guide.

./sbin/destroy-control-test.sh

Having a problem? If the problem is not obvious from the output (this is not a program usually oriented towards humans), the error and debug logs can be found like so:

nimbus $ ./sbin/cat-latest-logfile.sh

If putting the culprit error into a search engine does not produce any obvious answers, don't hesitate to contact the workspace-user mailing list. Describe your situation and any errors you are seeing. Attaching this debug log and including version information (of the VMM and libvirt and your OS) can often be helpful and reduce the amount of emails that need to occur.

Once you've gotten everything working, proceed to the next page where you will configure communication between the service node and the VMM node: SSH Setup.