Linux-KVM: Ubuntu 12.04 with Openvswitch
UPDATE: October 29, 2012
This article doesn’t work on Ubuntu 12.10, to use Ubuntu 12.10 please refer to my new article here.
Openvswitch is a better way of managing your virtual networking stacks for KVM and Xen. This can be used instead of the bridge-utils package, and has the ability to use VLANs, LACP, QoS, sFlow, and others. In this article we will be using Ubuntu 12.04 (beta1) to configure Openvswitch to hand out networking interfaces to KVM guests.
Please keep in mind that at the time of this writing Ubuntu 12.04 is beta, which means it is not fully baked for production. Though it is looking very stable.
Install Updates and Prerequisite Software
Check for updates, and install some commonly used hypervisor utilities along with the openvswitch and kvm stacks.
# apt-get update && apt-get dist-upgrade # apt-get install aptitude apt-show-versions ntp ntpdate vim kvm libvirt-bin vlan virtinst virt-manager virt-viewer openssh-server iperf pv openvswitch-controller openvswitch-brcompat openvswitch-switch openvswitch-datapath-source
Destroy the Default Libvirt Bridge (virbr0)
We want to delete the default libvirt interface which is created by default to be used for guests (I think it is NAT – I never use it).
# virsh net-destroy default # virsh net-autostart --disable default
Stop Libvirt and Qemu
This will prevent libvirt from bringing up the legacy bridge.
# service libvirt-bin stop # service qemu-kvm stop
Enable Openvswitch for Brcompat
# vi /etc/default/openvswitch-switch BRCOMPAT=yes
Purge Ebtables from System
Ebtables was needed when we were using VLANs on bridge-utils, we no longer need this for openvswitch.
# aptitude purge ebtables
Fix DKMS Module Build
When we originally installed openvswitch, you might have noticed an error talking about the module not available for your kernel, if you didn’t notice it you can see the same error by trying to start the openvswitch-switch service.
# service openvswitch-switch start
If you receive the module error, then we will need to build the openvswitch-datapath module.
# module-assistant auto-install openvswitch-datapath
Note that after installing new kernels you will most likely need to repeat this step, to rebuild the module.
Restart the Openvswitch Services
# service openvswitch-switch restart # service openvswitch-controller restart
Based on your reboot one of two things happened… Your modules came back up loaded and ready for business, though more likely they didn’t come up.
Here is an example of a working configuration. If you get this then move on to configuration of the switch itself.
# lsmod | grep brcom brcompat_mod 13512 0 openvswitch_mod 83993 2 brcompat_mod # service openvswitch-switch status ovsdb-server is running with pid 1885 ovs-vswitchd is running with pid 1908 ovs-brcompatd is running with pid 2096
# lsmod | grep brcom # service openvswitch-switch status ovsdb-server is running with pid 1885 ovs-vswitchd is running with pid 1908 ovs-brcompatd is running with pid 2096
Ensuring the Modules Load at Boot
The easy way to fix this is to restart the openvswitch-switch service whenever you reboot your hypervisor.
# service openvswitch-switch restart * Killing ovs-brcompatd (2096) * Killing ovs-vswitchd (1908) * Killing ovsdb-server (1885) * Starting ovsdb-server * Configuring Open vSwitch system IDs * Starting ovs-vswitchd * Starting ovs-brcompatd * iptables already has a rule for gre, not explicitly enabling
But that might not be ideal for your situation. As an alternative we can try and address the timing issue which is causing the whole issue in the first place. Basically what is happening is that upstart is starting openvswitch too late, and it is failing. If we adjust the waits in /etc/init/failsafe.conf
We want to change this…
$PLYMOUTH message --text="Waiting for network configuration..." || : sleep 40 $PLYMOUTH message --text="Waiting up to 60 more seconds for network configuration..." || : sleep 59 $PLYMOUTH message --text="Booting system without full network configuration..." || :
$PLYMOUTH message --text="Waiting for network configuration..." || : sleep 1 $PLYMOUTH message --text="Waiting up to 60 more seconds for network configuration..." || : sleep 1 $PLYMOUTH message --text="Booting system without full network configuration..." || :
Now after a reboot you shouldn’t have this problem with the modules not loading.
# lsmod | grep brcom brcompat_mod 13512 0 openvswitch_mod 83993 1 brcompat_mod
Design our Network Layout
In our example we will use 2 physical interfaces, one of which (eth0) we will consider our management interface, it will have a static IP address and will be “the hypervisor” the other one (eth1) will be the uplink for our Openvswitch. Now eth1 will have an untagged fake bridge (br0) which will be the parent for our other fake bridge (br10) which will be tagged 10. If we had more to add they would still use br0 as the parent and then be tagged on the subordinate fake bridge. Another important thing to keep in mind is that your uplinked switch port must support and be configured to allow your devices to tag for these other VLANs.
Configure Openvswitch Fake Bridges
# ovs-vsctl add-br br0 # ovs-vsctl add-port br0 eth1 # ovs-vsctl add-br br10 br0 10 # ovs-vsctl list-br br0 br10
Create Dummy Interface Entries
We need to edit /etc/network/interfaces to add some entries
# cat /etc/network/interfaces auto eth0 iface eth0 inet static address 10.0.0.141 netmask 255.255.255.0 gateway 10.0.0.1 auto eth1 iface eth1 inet manual up ifconfig $IFACE 0.0.0.0 up down ifconfig $IFACE down auto br0 iface br0 inet manual up ifconfig $IFACE 0.0.0.0 up down ifconfig $IFACE down auto br10 iface br10 inet manual up ifconfig $IFACE 0.0.0.0 up down ifconfig $IFACE down
These bring up the interfaces with No IP address, now if you run ifconfig -a they should be visible.
Once you have a guest up on br0 or br10 you will see a “vnet0” interface.