Xen (not XeServer, btw!) seems to have taken a bak-seat recently, RedHat/CentOS/Fedora concentrating on KVM and Debian silently neglecting it.

This is reflected in documentation, there is a lot of outdated stuff around, especially about bridged setups. Same occurs to packages, at least in Debian Wheezy (NB: I also tried on testing, same results with fairly newer packages).

My aim was a virtual host which is directly connected to the internet without any external firewall running different virtual machines which ARE thoroughly firewalled. In order to archive this, I am running the quite decent Sophos UTM (formerly Astaro) as a VM, this is the only virtual machine with direct access to the external network interface. It’s other interface just like all other VMs are connected to an internal bridge without any link to the rest of the world. This is why routing isn’t an option.

This article focusses on the Xen server and the bridging setup, maybe I will write another one later about Sophos UTM etc.

 

I am running this setup on some servers at Hetzner, though this should be working at most other hosters (some tend to drop the switch connection when they sense a pseudo ARP-spoofing, take care!), I am in no way affiliated to Hetzner.

My setup needs a secondary IP address (the main IP address is used for management of the host, I am assuming the following setup:

External Host-IP: 1.2.3.4

Secondary IP, used on UTM: 1.2.3.6. At least at Hetzner, this IP address needs to have it’s own MAC address assigned, this can be done in their Robot tool.

Setting up the host

I want to have my host running directly on the (Software-)RAID, I personally don’t really like running the OS on LVM. But I also want to have my VMs live in an LVM realm in order to easily take snapshots, clone etc.

This means that Hetzner’s default setup isn’t very helpful. But they have an answer file based installation using the rescue system. Therefore: boot into the rescue system and run install image.

Note: Preserve the temporary password for the rescue system, you will need it for the freshly installed system!

This lets you define your custom install file and then installs everything within a few minutes. I chose Debian Wheezy Minimal. The only two settings I changed were the hostname (I am using dome in this example) and the partition setup:

I chose 50GB for my root filesystem and 12GB swap.

After saving the file and starting the installation, I had to wait for about five minutes and was presented with a brand new Debian system.

After the first boot I changed the root password and added my own SSH key.

Note: This document doesn’t cover hardening your server, which you really should do!

First thing to do is updating all package sources:

root@dom2 ~ # apt-get update

I tend to install emacs23-nox as soon as possible, YMMV.

It is quite handy to add your domain, if you are using one, to /etc/resolv.conf and to /etc/hosts.

Next is changing the network setup, so edit /etc/network/interfaces :

# Loopback device:
auto lo
iface lo inet loopback

# device: eth0 => transfigured to a bridge!
auto  virbr0
iface virbr0 inet static
  address   1.2.3.4
  broadcast 1.2.3.191
  netmask   255.255.255.192
  gateway   1.2.3.129
  bridge_ports eth0
  bridge_stp off
  bridge_fd 1
  bridge_hello 2
  bridge_maxage 12
  allow-hotplug virbr0

auto virbr1
iface virbr1 inet static
  address 192.168.101.1
  netmask 255.255.255.0
  bridge_ports none
  bridge_fd 1
  bridge_stp off
  bridge_hello 1
  down ifconfig virbr1 down
  allow-hotplug virbr1

So we transformed our (only) network card eth0 into a bridge called virbr0 and added a secondary bridge, virbr1.

Set up Xen

First install the xen system (4.1 on Wheezy) and the xen-tools which are quite helpful setting up VMs.

root@dom2 ~ # apt-get install xen-system-amd64
root@dom2 ~ # apt-get install xen-tools

This will install xen and all the necessary tools.

In order to boot into a xen enabled hypervisor, we need to adapt GRUB:

root@dom2 ~ # dpkg-divert --divert /etc/grub.d/08_linux_xen --rename /etc/grub.d/20_linux_xen

Before we reboot we also adapt the boot command line in /etc/default/grub :

GRUB_CMDLINE_LINUX_DEFAULT="nomodeset dom0_max_vcpus=1 dom0_vcpus_pin dom0_mem=2048M nopat cgroup_enable=memory swapaccount=1"

This basically limits resources on Dom0.

Now update grub and reboot:

root@dom2 ~ # update-grub
Generating grub.cfg ...
Found linux image: /boot/vmlinuz-3.2.0-4-amd64
Found initrd image: /boot/initrd.img-3.2.0-4-amd64
Found linux image: /boot/vmlinuz-3.2.0-4-amd64
Found initrd image: /boot/initrd.img-3.2.0-4-amd64
done
root@dom2 ~ # reboot

After reboot we can check if Xen is up and running.

root@dom2 ~ # xm list
Name                                        ID   Mem VCPUs      State   Time(s)
Domain-0                                     0  2047     1     r-----      7.3

Looks fine.

Now we need to set up the network, which is quite straight forward:

Edit the file /etc/xen/xend-config.sxp  and comment out everything about networking, routing and vif except this line:

(vif-script vif-bridge)

You  may also fine tune your Xen setup by changing the following lines:

# enable-dom0-ballooning below) and for xm mem-set when applied to dom0.
(dom0-min-mem 196)

# Whether to enable auto-ballooning of dom0 to allow domUs to be created.
# If enable-dom0-ballooning = no, dom0 will never balloon out.
(enable-dom0-ballooning no)

# xen kernel to reserve the memory for 32-bit paravirtual domains, default
# is "0" (0GB).
#(total_available_memory 0)

# In SMP system, dom0 will use dom0-cpus # of CPUS
# If dom0-cpus = 0, dom0 will take all cpus available
(dom0-cpus 1)

The first thing we changed tells Xen to run a script called vif-bridge  located in /etc/xen/scripts/  as soon as a virtual machine is being created. The script basically checks if the bridge exists and connects the VMs virtual network card to the bridge.

Now we need to adapt this file to our naming convention, so let’s replace the occurrences of xenbr  to virbr  in the file /etc/xen/scripts/vif-bridge :

  # This lets old config files work without modification
  #
  if [ ! -e "/sys/class/net/$bridge" ] && [ -z "${bridge##virbr*}" ]
  then
     if [ -e "/sys/class/net/eth${bridge#virbr1}/bridge" ]
     then
        bridge="eth${bridge#virbr1}"
     fi
  fi
fi

Now restart xend (for some reason the service is called xen  on Debian.

root@dom2 ~ # service xen restart
[ ok ] Restarting Xen daemons: xend xend xenconsoled.

Getting the first VM up and running

Using xen-create-image  from the xen-tools makes it a piece of cake installing our first VM:

root@dom2 ~ # xen-create-image --hostname=test --ip=192.168.101.200 --netmask 255.255.255.0 --gateway=192.168.101.1 --bridge=virbr1 --lvm=vg0 --mirror=http://mirror.hetzner.de/debian/packages --memory 512m --swap 1000M --dist=wheezy

WARNING
-------

  You appear to have a missing vif-script, or network-script, in the
 Xen configuration file /etc/xen/xend-config.sxp.

  Please fix this and restart Xend, or your guests will not be able
 to use any networking!


General Information
--------------------
Hostname       :  test
Distribution   :  wheezy
Mirror         :  http://mirror.hetzner.de/debian/packages
Partitions     :  swap            1000M (swap)
                  /               4Gb   (ext3)
Image type     :  full
Memory size    :  512m
Kernel path    :  /boot/vmlinuz-3.2.0-4-amd64
Initrd path    :  /boot/initrd.img-3.2.0-4-amd64

Networking Information
----------------------
IP Address 1   : 192.168.101.200 [MAC: 00:16:3E:88:89:EC]
Netmask        : 255.255.255.0
Gateway        : 192.168.101.1


Creating swap on /dev/vg0/test-swap
Done

Creating ext3 filesystem on /dev/vg0/test-disk
Done
Installation method: debootstrap
Done

Running hooks
Done

No role scripts were specified.  Skipping

Creating Xen configuration file
Done

No role scripts were specified.  Skipping
Setting up root password
Generating a password for the new guest.
All done
Logfile produced at:
         /var/log/xen-tools/test.log

Installation Summary
---------------------
Hostname        :  test
Distribution    :  wheezy
IP-Address(es)  :  192.168.101.200
RSA Fingerprint :  76:ab:f1:50:4e:71:49:7e:06:13:87:5c:8a:1d:62:82
Root Password   :  XXXXXXXXXX

You can safely ignore the warning about vif-bridge.

Now there’s a little bug in the xen-tools:

root@dom2 ~ # xm create /etc/xen/test.cfg
Using config file "/etc/xen/test.cfg".
Error: invalid literal for int() with base 10: '512m'

So edit /etc/xen/test.cfg and remove the m from 512m:

memory      = '512m'

memory      = '512'

Now let’s run it:

root@dom2 ~ # xm create /etc/xen/test.cfg
Using config file "/etc/xen/test.cfg".
Started domain test (id=6)

We can now connect a console to the vm and see why’s going on (you can also create it with the -c parameter above …).

root@dom2 ~ # xm console test

Debian GNU/Linux 7 test hvc0

test login: root
Password:
Linux test 3.2.0-4-amd64 #1 SMP Debian 3.2.60-1+deb7u3 x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
root@test:~#

Hint: CTRL + 5 gets you to of the console again.