Install VMM Software

On this page, you will install the Nimbus VMM software workspace-control and any of its dependencies (including Xen or KVM as well as libvirt, if these are not present already).

workspace-control contains scripts that will help you test that your Xen/KVM/libvirt installation will work with the programmatic access patterns that Nimbus uses.

You should use one VMM node for this exercise. When things are working end to end, you can propagate the configuration and dependencies to all of the other nodes on your cluster (assuming that they are roughly homogeneous).

Dependencies (#)

Before automating a VMM node with Nimbus software, it needs to be a VMM node. So the first thing we need to do is make sure that all the virtualization and networking dependencies are present.

The workspace service currently manages Xen or KVM VMs. It interfaces with both of these systems using a library called libvirt.

This guide will explain how to install the Nimbus workspace control software on top of both Xen 3.x and KVM. The instructions are intermingled so it is recommended that you first read our notes page for the hyper-visor that you intend to use:

• Xen
• KVM

There are many steps in this section which are common to both KVM and Xen. However there are also some slight differences mixed in with each step. We will visually distinguish the two with the following styles.

The reader can then key in on the information specific to their cloud.

Basic Dependencies

Install Xen 3.x or KVM as described in their respective notes above.

Install ebtables using your Linux distribution's package management tools.

Install libvirt version 0.7 or higher using your Linux distribution's package management tools and user guides.

Make sure you have Python 2.4 or higher (but not Python 3.x).

If your Linux distribution doesn't support these packages well, you can try installing from source or binary, starting at libvirt.org (and either xen.org or linux-kvm.org) and ebtables.

Our next step in this guide will be to create a unix account for workspace-control and then install it. This will give us access to some scripts we can use to further verify the system is ready to be automated with Nimbus.

Choose/Create Privileged User Account (#)

There are two system accounts needed on the VMMs.

• privileged account - Pick an account to make privileged (using the sudo program).

  In this guide (especially in the command samples) we will refer to an account of this type named nimbus with terminal prompts like "nimbus $"

  Note: this is not a privileged account in the sense that root is a privileged account. This is a regular account that you will give powers to during the setup process. It is an account that will be privileged because it will have control over the hyper-visor.

• superuser account - The root account is necessary to install dependencies on the VMM nodes (Xen/KVM, ebtables, etc.) and also to install the Nimbus agent that lives on the VMM nodes (workspace control).

  In this guide (especially in the command samples) we will refer to an account of this type named root with terminal prompts like "root #"

  Note that root is not needed on the service node, just here on the VMM.

Set up libvirt permissions (#)

OK, you've got your non-root user nimbus created, now we need to allow it to control VMs using libvirt.

First, try out a libvirt command as root (it will not change anything if it succeeds or fails).

root # virsh -c 'xen+unix:///' list

root # virsh -c 'qemu:///system' list

Once it works as root, try as the nimbus user:

nimbus $ virsh -c 'xen+unix:///' list

nimbus $ virsh -c 'qemu:///system' list

You may hit an error about failing to connect to the hypervisor.

error: unable to connect to '/var/run/libvirt/libvirt-sock': Permission denied
error: failed to connect to the hypervisor

Why? Because you are not root. Check out the permissions on /var/run/libvirt/libvirt-sock

root # ls -l /var/run/libvirt/libvirt-sock
srwx------ 1 root root 0 Jul  1 21:28 /var/run/libvirt/libvirt-sock

The solution is to set the domain socket with certain permissions.

user = "root"
group = "root"
dynamic_ownership = 0
clear_emulator_capabilities = 0

user = "nimbus"
group = "nimbus"
dynamic_ownership = 0

First, make the nimbus user a member of the libvirtd group:

root # usermod -a -G libvirtd nimbus

Make the libvirt domain socket available for reading and writing by the libvirtd group by enabling the following configurations in /etc/libvirt/libvirtd.conf

unix_sock_group = "libvirtd"
unix_sock_ro_perms = "0770"
unix_sock_rw_perms = "0770"
unix_sock_dir = "/var/run/libvirt"

On some distributions you may need to specify auth_unix_rw = "none" if libvirt defaults to using PolicyKit.

Make sure the group change has taken effect with the nimbus user (using the "groups" command within the shell, you may need to relogin).

Try again:

nimbus $ virsh -c 'xen+unix:///' list

nimbus $ virsh -c 'qemu:///system' list

If you can still not do this basic list command as the nimbus user, look into your distribution's libvirt configurations and also don't hesitate to contact the workspace-user mailing list with problems.

Install workspace-control (#)

We can now download the VMM control package to the machine.

Download the "Control Agents" tar file from the download page, and untar it. This archive contains both workspace-control and the workspace pilot. For this configuration we are using workspace-control. You need to copy it to the /opt/nimbus destination directory like so:

root # curl -O https://www.nimbusproject.org/downloads/nimbus-iaas-controls-2.10.1.tar.gz
root # tar xfz nimbus-iaas-controls-2.10.1.tar.gz
root # mv nimbus-iaas-controls-2.10.1/workspace-control/* /opt/nimbus

Make sure the directory hierarchy is as the default configurations -- and this guide -- expect.

root # [ -f /opt/nimbus/bin/workspace-control.sh ] && echo "Correct."
Correct.

Now make sure the permissions are set up for running via sudo safely. Note the chown -R nimbus part, this is crucial for letting workspace-control do a lot of its work as the non-root user nimbus (only calling out to root via sudo to perform specific priviliged tasks).

root # cd /opt/nimbus
root # chown -R root bin etc lib libexec src
root # chown -R nimbus var
root # find . -type d -exec chmod 775 {} \;
root # find . -type f -exec chmod 664 {} \;
root # find bin sbin libexec -iname "*sh" -exec chmod 755 {} \;

Test the basic dependencies

Before moving on with a VM launching test, first we should confirm the basic assumptions about the environment. The workspace-control package contains a script that will examine the system.

Run the following command:

nimbus $ ./sbin/test-dependencies.sh

If this script reports you have a problem, look into your distribution's libvirt support (especially the python bindings that were discussed earlier) and don't hesitate to contact the workspace-user mailing list with problems.

Pick the kernel to use (#)

nimbus $ KERNELDIR=/opt/nimbus/var/workspace-control/kernels
nimbus $ cp /boot/vmlinuz-2.6.16.29-xen $KERNELDIR/default
nimbus $ cp /boot/initrd-2.6.16.29-xen $KERNELDIR/default-initrd

Obtain the sample image (#)

A sample image has been prepared for testing. This is a basic VM that the Nimbus developers are familiar with, meaning it will be easier to get support help using this VM.

nimbus $ cd /tmp
nimbus $ curl -O https://www.nimbusproject.org/downloads/nimbus-z2c.gz
nimbus $ gunzip nimbus-z2c.gz

nimbus $ cd /tmp
nimbus $ curl -O https://www.nimbusproject.org/downloads/ubuntu10.10.gz
nimbus $ gunzip ubuntu10.10.gz

Configure libvirt hypervisor (#)

Workspace control must be configured to use the correct libvirt hypervisor. This is done in /opt/nimbus/etc/workspace-control/libvirt.conf. The default hypervisor is Xen. If you are using KVM you'll need to edit the vmm property in this file.

[libvirt]
vmm: kvm0

Configure default bridge (#)

You set up Xen/KVM to use bridged networking, now it is time to check what the actual name of the bridge will be that the guest VM network card uses to get traffic to the LAN.

Using the brctl tool, you will see something like this in the list of bridges on the VMM system.

root # brctl show
bridge name     bridge id               STP enabled     interfaces
xenbr0          8000.feffffffffff       no              peth0

# brctl  show
bridge name     bridge id               STP enabled     interfaces
virbr0          8000.000000000000       yes

Configure sudo and ebtables script (#)

Before we can do a live test, we need to configure workspace-control to use sudo privileged properly.

Using the visudo command, add the necessary sudo policies. These policies reflect need to use the user that will be running workspace-control ("nimbus") and the correct full paths to the libexec tools. See /opt/nimbus/etc/workspace-control/sudo.conf for all the details and sample rules.

Make sure the sudo rules work without a password as the nimbus user.

nimbus $ /usr/bin/sudo /opt/nimbus/libexec/workspace-control/xen-ebtables-config.sh
ERROR: requires at least 2 arguments, syntax: add|rem <vifname> [<macaddr> <ipaddr> [<dhcpif>]]

nimbus $ /usr/bin/sudo /opt/nimbus/libexec/workspace-control/kvm-ebtables-config.sh
ERROR: requires at least 2 arguments, syntax: add|rem <vifname> [<macaddr> <ipaddr> [<dhcpif>]]

Seeing "ERROR" is a good thing here. If you see a password prompt, type in the nimbus user's password which hopefully is a "first time use" warning from sudo. Then try again. The goal is to perform this command from the nimbus account without needing any password.

You may need to comment out any "requiretty" setting in the sudoers policy, like so:

#Defaults    requiretty

The commands run via sudo are not using a terminal and so if you have "requiretty" enabled, this can cause a failure.

Obtain the workspace-control network sample (#)

In a previous section of this guide, you configured DHCPd and the network addresses to give to guest VMs.

The central IaaS services container wrote out the dhcpd.entries file that will be used for the DHCPd server configuration. In the same directory a sample file for workspace-control was written out, we will use this to test a VM launch via libvirt with your DHCPd server running. The file you need to obtain is called control.netsample.txt

Transfer this file from the central node to anywhere on the VMM node: $NIMBUS_HOME/services/var/nimbus/control.netsample.txt

We will use it in the next step. The example location on the VMM will be /tmp/control.netsample.txt.

Test VM creation with a generated libvirt config (#)

Once you have obtained the control.netsample.txt file, have the kernel configured and have the default bridge configured, you are ready to try starting the test VM using a libvirt configuration file.

This is not just any libvirt configuration file, but one representing exactly the configuration that the workspace-control tool will use to launch VMs on your system automatically.

A script called libvirt-xml.sh lives in /opt/nimbus/sbin that will generate XML that we can use to launch VMs using the libvirt commandline tool called virsh

It takes a few arguments:

nimbus $ cd /opt/nimbus
nimbus $ ./sbin/libvirt-xml.sh -h
Usage: see help (-h).

Options:
  --version         show program's version number and exit
  -h, --help        show this help message and exit
  --image=PATH      VM image file to test
  --netsample=PATH  Net sample file obtained from service node
  --memory=INT      Amount of MB RAM to assign
  --mountpoint=id   Mountpoint like 'sda1' to mount image to

That image we downloaded is at /tmp/nimbus-z2c or /tmp/ubuntu10.10, that will be the value for the --image flag.

The --memory value will be something easily doable by your system, e.g. 256.

The --mountpoint value will be sda1 if you are using xen with the nimbus-z2c image and hda if you are using kvm with the ubuntu10.10 image.

That leaves the --netsample flag. The value for this will be the path to the control.netsample.txt file we just obtained in the last step.

Run the following:

nimbus $ ./sbin/libvirt-xml.sh  --image /tmp/nimbus-z2c --netsample \
/tmp/control.netsample.txt --memory 256 --mountpoint sda1

nimbus $ ./sbin/libvirt-xml.sh  --image /tmp/ubuntu10.10 --netsample \
/tmp/control.netsample.txt --memory 256 --mountpoint hda

On stdout, you will see an XML document printed.

Now run the following script to reset things:

nimbus $ ./sbin/destroy-control-test.sh

This may need to be repeated to solve any problems that are being mentioned. Anytime you run libvirt-xml.sh, you should pair it with destroy-control-test.sh before trying again.

Having a problem? If the problem is not obvious from the output, the error and debug logs can be found like so:

nimbus $ ./sbin/cat-latest-logfile.sh

If putting the culprit error into a search engine does not produce any obvious answers, don't hesitate to contact the workspace-user mailing list. Describe your situation and any errors you are seeing. Attaching this debug log and including version information (of the VMM and libvirt and your OS) can often be helpful and reduce the amount of emails that need to occur.

Once you have the XML getting printed, capture it to a file:

nimbus $ ./sbin/libvirt-xml.sh --image /tmp/nimbus-z2c --netsample \
/tmp/control.netsample.txt --memory 256 --mountpoint sda1 > /tmp/z2c.xml

nimbus $ ./sbin/libvirt-xml.sh --image /tmp/ubuntu10.10 --netsample \
/tmp/control.netsample.txt --memory 256 --mountpoint hda > /tmp/z2c.xml

Make sure the /tmp/z2c.xml file is what we expect. It should start with lines like this:

<?xml version="1.0" encoding="UTF-8" standalone="no" ?>
<domain type='test'>
    <name>control-test</name>

OK! Now as the nimbus user run the virsh command to boot up the VM.

nimbus $ virsh -c 'xen+unix:///' create /tmp/z2c.xml

nimbus $ virsh -c 'qemu:///system' create /tmp/z2c.xml

Now the VM will boot. If it gets the correct network address from DHCP you should be able to ping it and even log in over SSH.

First find the IP address we are expecting it to get:

nimbus $ grep ip /tmp/control.netsample.txt | awk '{print $2}'

Then give it a ping:

nimbus $ ping -c 1 192.168.0.2

If this does not work, the next step will be to log in via the console and troubleshoot what happened. You can log in using console to the nimbus-z2c image using the root account with password root

nimbus $ virsh -c 'xen+unix:///' console control-test

nimbus $ virsh -c 'qemu:///system' console control-test

Does it not get an IP address from DHCPd? Is DHCPd running?

Did it get some other IP address from DHCPd? Your DHCPd may be misconfigured, perhaps it is not giving specific MAC addresses specific IP addresses. Perhaps you did not load it with the current dhcpd.leases entries from Nimbus.

If you get an error like "There is a VM running with this handle but there is no record of it in the program" then you may have been starting with virsh but not through libvirt-xml.sh. The way to correct this is to run virsh destroy control-test directly.

Seeing an error from a component, for example in the libvirt logs? If putting the error into a search engine does not produce any obvious answers, don't hesitate to contact the workspace-user mailing list. Describe your situation and any errors you are seeing. Including version information (of the VMM and libvirt and your OS) can often be helpful and reduce the amount of emails that need to occur.

When all is well, you should be able to log in as root over SSH. Trying this from off the node is the best final test.

ssh [email protected]

Final test: VM creation using workspace-control (#)

The final test will be to launch the same file with the same parameters, but using workspace-control.sh just as the service will. We will use a wrapper program called control-test.sh to invoke workspace-control.sh.

If you have a control-test VM running at the moment, shut it down using the kill script:

./sbin/destroy-control-test.sh

Gather the same arguments you used for ./sbin/libvirt-xml.sh

$ ./sbin/control-test.sh --image /tmp/nimbus-z2c --netsample \
/tmp/control.netsample.txt --memory 256 --mountpoint sda1

$ echo $?
0

$ ./sbin/control-test.sh --image /tmp/ubuntu10.10 --netsample \
/tmp/control.netsample.txt --memory 256 --mountpoint hda

$ echo $?
0

If the exit code is zero, we are good to go. SSH again to test for sure. Run the destruction script and you can move along to the next section in this guide.

./sbin/destroy-control-test.sh

Having a problem? If the problem is not obvious from the output (this is not a program usually oriented towards humans), the error and debug logs can be found like so:

nimbus $ ./sbin/cat-latest-logfile.sh

If putting the culprit error into a search engine does not produce any obvious answers, don't hesitate to contact the workspace-user mailing list. Describe your situation and any errors you are seeing. Attaching this debug log and including version information (of the VMM and libvirt and your OS) can often be helpful and reduce the amount of emails that need to occur.

Once you've gotten everything working, proceed to the next page where you will configure communication between the service node and the VMM node: SSH Setup.