Linux KVM: Ubuntu 12.10 with Openvswitch

October 24th, 2012 | Tags: , , , , ,

Today I am revisiting my previous post on Openvswitch on Ubuntu 12.04.  Things have changed since then.  Previously Openvswitch was relatively new and as such the userland tools (with libvirt being the one I use) didn’t support it yet, so while you could have used Openvswitch by executing the kvm processes for each VM manually.  Since this was a predictable problem the Openvswitch guys created a brcompat module, which would take the place of the bridge_utils functionality and expose Openvswitch bridges as legacy bridges.  This allowed the existing userland which already had the legacy bridge functionality builtin to continue to operate the same way.  Anyways as of libvirt 0.9.11 the native Openvswitch functionality is present and supported.  As such I am updating my documentation to include this alternate and preferred method of operation.

Libvirt Versions

Here is the libvirt version present in Ubuntu 12.10.

# libvirtd --version
libvirtd (libvirt) 0.9.13

Here is the libvirt version present in Ubuntu 12.04.

# libvirtd --version
libvirtd (libvirt) 0.9.8

So as we can see above this guide will only work on Ubuntu 12.10 and not on 12.04 or older.  For Ubuntu 12.04, please refer to my older article available here.

Install Updates and Prerequisite Software

Check for updates, and install some commonly used hypervisor utilities along with the openvswitch and kvm stacks.  Note that below I am installing the openvswitch-brcompat package.  I do not think that you need this, however I did install it in my environment in case I was going to need to fail back to the brcompat route, and I haven’t had the time to validate it without that package installed.  But it is not being used in this configuration.

# apt-get install aptitude apt-show-versions ntp ntpdate vim kvm libvirt-bin vlan virtinst virt-manager virt-viewer openssh-server iperf pv openvswitch-controller openvswitch-brcompat openvswitch-switch

Destroy the Default Libvirt Bridge (virbr0)

We want to delete the default libvirt interface which is created by default to be used for guests (I think it is NAT – I never use it).

# virsh net-destroy default
# virsh net-autostart --disable default

Stop Libvirt and Qemu

This will prevent libvirt from bringing up the legacy bridge.

# service libvirt-bin stop
# service qemu-kvm stop

Purge Ebtables from System

Ebtables was needed when we were using VLANs on bridge-utils, we no longer need this for openvswitch.

# aptitude purge ebtables

Restart the Openvswitch Services

#service openvswitch-switch restart
#service openvswitch-controller restart

Configure Interfaces

Now this configuration is a single interface which will be used both for management of the hypervisor as well as the bridge.  There are three key things to note.  We need to set the physical interface to “manual” (instead of “dhcp” or “static”).  We also need to perform a quick up and down of the physical interface in order to be able to initialize the bridge on it.  Finally we need to setup the configuration of an interface which will connect up to the bridge that we define, this becomes the management interface for the OS.  In this example we are simply using DHCP.  This will show us if it is working quickly, but it can also add some complexity if things are not working, please keep in mind that you might want to just use a static IP address to eliminate that as an issue.

# cat /etc/network/interfaces
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet manual
up ifconfig $IFACE 0.0.0.0 up
down ifconfig $IFACE down

auto ovsbr0p1
iface ovsbr0p1 inet dhcp

Configure our Network

Configuring the network will be a three step process.  First we define the bridge which I am naming ovsbr0 unlike legacy bridges we don’t need to stick to brX as a naming convention.

# ovs-vsctl add-br ovsbr0

Second we connect our bridge with its uplink which in this case is eth0.

# ovs-vsctl add-port ovsbr0 eth0

Finally we create a port for our hypervisor to connect up to the bridge and out the physical interface.

# ovs-vsctl add-port ovsbr0 ovsbr0p1 -- set interface ovsbr0p1 type=internal

After all of that we should end up with something like below.

# ovs-vsctl show
ed1986fc-830e-4f66-9ce7-92fad0787bb8
Bridge "ovsbr0"
Port "ovsbr0"
Interface "ovsbr0"
type: internal
Port "ovsbr0p1"
Interface "ovsbr0p1"
type: internal
Port "eth0"
Interface "eth0"
ovs_version: "1.4.3"

Reboot

# reboot

Adjusting Service Sleeps

If your networking seems to hang on boot up.

In /etc/failsafe.conf we need to adjust the sleeps.

Change this…

$PLYMOUTH message --text="Waiting for network configuration..." || :
sleep 40
$PLYMOUTH message --text="Waiting up to 60 more seconds for network configuration..." || :
sleep 59
$PLYMOUTH message --text="Booting system without full network configuration..." || :

To this…

$PLYMOUTH message --text="Waiting for network configuration..." || :
sleep 1
$PLYMOUTH message --text="Waiting up to 60 more seconds for network configuration..." || :
sleep 1
$PLYMOUTH message --text="Booting system without full network configuration..." || :

Obviously if you end up needing this change, make sure you reboot again to ensure that it resolved the issue.

Define VM Configuration

I am not going to go into details on how to create a guest from XML, but if you need some help on that refer to my previous article here.

Basically below we have a snippet from a guests XML configuration which configures a single network interface.  You can edit an existing machine configuration using “virsh edit vmnamehere”

<interface type='bridge'>
<mac address='52:54:00:0f:b4:7a'/>
<source bridge='ovsbr0'/>
<virtualport type='openvswitch'/>
<model type='virtio'/>
</interface>

After you have merged the changes in, libvirt will go create a port on Openvswitch for this guest.  Which we can see below as vnet0.

# ovs-vsctl show
ed1986fc-830e-4f66-9ce7-92fad0787bb8
Bridge "ovsbr0"
Port "ovsbr0"
Interface "ovsbr0"
type: internal
Port "ovsbr0p1"
Interface "ovsbr0p1"
type: internal
Port "vnet0"
Interface "vnet0"
Port "eth0"
Interface "eth0"
ovs_version: "1.4.3"

Additionally we can see that in our VM configuration libvirt has added more details, the target dev and parameters interfaceid.  This is how the VM knows which port on the Openvswitch to connect to.

<interface type='bridge'>
<mac address='52:54:00:0f:b4:7a'/>
<source bridge='ovsbr0'/>
<virtualport type='openvswitch'>
<parameters interfaceid='a78ac510-2442-1326-c60f-d76c1e830bd6'/>
</virtualport>
<target dev='vnet0'/>
<model type='virtio'/>
<alias name='net0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>

That is all there is to it.  When you bring up your guest. you will have networking exactly as you expected.  But now you open yourself up to a world of possibilities with Openvswitch, such as port mirroring, QoS, LACP and many more.

  1. Mark Garber
    November 10th, 2012 at 11:34
    Quote | #1

    You might add after ‘apt-get…’ that users need to add themselves to the libvirtd group ie
    sudo adduser `id -un` libvirtd. After doing a fresh install I had to do this to keep the os from complaining. After doing several installs to get ovs and vm to work together seamlessly, yours was the best howto on this. Thanks:)

    • matthew.mattoon
      November 26th, 2012 at 23:32
      Quote | #2

      Hi Mark,

      In this article I am assuming that folks already have a working KVM + libvirt environment and that they are simply trying to expand functionality by integrating openvswitch into that environment.

      -matt

  2. reever
    November 13th, 2012 at 09:24
    Quote | #3

    I think too !

    very best howto on the planet at this time

  3. Yitzhak Bar Geva
    November 26th, 2012 at 16:59
    Quote | #4

    My horror story:
    (Which doesn’t detract one iota from my genuine appreciation of this excellent post)

    I started plugging in the commands one after the other on my remote dedicated Ubuntu 12.10 server.
    Got here:
    # ovs-vsctl add-port ovsbr0 eth0
    # ovs-vsctl add-port ovsbr0 ovsbr0p1 — set interface ovsbr0p1 type=internal
    Pressed “enter” and Boom(!), lost communication and got locked out (see below)

    Researched by going through the same motions on my local machine. Same deal. Debugging revealed that some of the ARP packets send to the machine were being forwarded through eth0 on to ovsbr0 and the kernel’s ARP table was corrupt.
    But get this: Disabling ovs-vsctl (ifdown and removing it from /etc/network/interfaces) didn’t make a dent. Neither did stopping openvswitch libvirt-bin and qemu-kvm services.
    Only ovs-vsctl del-br ovsbr0 restored the flow of arp packets to eth0.

    Question:
    Granted, I had not yet defined a VM, but still, my question is how can we control this?
    Thanks for the fine work. Pointers appreciated,
    Yitzhak

    Not precisely our problem, but similar:
    http://www.linuxfoundation.org/collaborate/workgroups/networking/bridge#No_traffic_gets_trough_.28except_ARP_and_STP.29

    P.S. (from above) Normal mortals don’t attempt tricky things like fiddling with bridging on remote servers, but in this case, it’s mine easy enough access to “clean up the mess”

    • matthew.mattoon
      November 26th, 2012 at 23:28
      Quote | #5

      Hi Yitzhak,

      I too ran into that, and in my previous iteration of the article for Ubuntu 12.04 I used a work around of having a separate interface which was not a part of the openvswitch configuration. This was more due to the requirement of running the DKMS Fix after installation of new kernels. Since 12.10 no longer requires the BRCOMPAT module, we no longer have to deal with that instability. So once you have it up and configured then it will stay up and configured. Now if you have no choice but to make this change remotely then I would script it…

      I would probably make the /etc/network/interfaces file changes manually (as if openvswitch was already configured) then script all of the ovs-vsctl steps (create ovsbr0, attach it to eth0, create ovsbr0p1) and as the final step in the script restart networking. Please keep in mind I have not tested the steps in this comment, so make sure you (or anyone else reading this) test your script for the desired effect on a machine which you have alternate mechanisms to connect to (out-of-band management (ilom, drac, ip kvm, etc) or a separate network interface outside of openvswitch.

      Now as for what you mentioned about your repro steps on your local machine. It sounds like you might have had some mistakes in your /etc/network/interfaces file. Were you trying to define an address on eth0, or did you have it in a mode other than manual (static or dhcp). If so that would break your openvswitch config, as eth0 cannot be used as a normal interface in this config since it is the uplink for the switch.

      -matt

  4. Yitzhak Bar Geva
    November 28th, 2012 at 04:46
    Quote | #6

    Here’s what I had to do to get it to work after every boot. The ovs-vsctl configurations done once and stayed up. What is shown is run after networking came up with /etc/network/interfaces as shown in pinterfaces:

    cd /etc/network
    service networking stop
    cp qinterfaces interfaces
    service networking start
    cp pinterfaces interfaces
    service networking restart

    Where qinterfaces:
    auto eth0
    iface eth0 inet manual
    up ifconfig $IFACE 0.0.0.0 up
    down ifconfig $IFACE down

    auto ovsbr0p1
    iface ovsbr0p1 inet dhcp

    and pinterfaces (the system default):
    auto lo
    iface lo inet loopback

    auto eth0
    iface eth0 inet static
    address 10.0.0.110
    netmask 255.255.255.0
    network 10.0.0.0
    broadcast 10.0.0.255
    gateway 10.0.0.138
    dns-nameservers 212.117.129.3 212.117.128.6
    dns-search domain.local

    Interestingly, I couldn’t convince ovsbr0p1 to take a static IP. It only works with dhcp, with an incremented IP following every boot – as if the previous IP’s are still in use.
    Advice warmly accepted.
    Thanks again,
    Yitzhak

  5. Yitzhak Bar Geva
    November 29th, 2012 at 10:22
    Quote | #7

    Did something stupid. Skip what I wrote above. I can explain via e-mail
    Yitzhak

  6. Andrea
    December 4th, 2012 at 08:14
    Quote | #8

    Hello.
    Could I try this howto article using an Ubuntu 12.10 virtual machine running on VMware Workstation?
    Thanks.
    Andrea

    • matthew.mattoon
      December 4th, 2012 at 11:04
      Quote | #9

      This article covers using openvswitch with KVM. KVM will not work within VMWare workstation as it requires access to VT on the chip. So could you configure openvswitch with a VMWare guest, yes. But I don’t think this would be worth much except for familiarizing yourself with the process.

      • Calreth
        February 19th, 2013 at 14:55

        It actually is possible as you can configure VMWare to enable VT-x in the client, thus giving the client access to the virtualization hardware. I’ve been able to run nested VMs this way.

        • matthew.mattoon
          February 19th, 2013 at 17:07

          Calreth,

          It is possible yes, but with far too many caveats to qualify in a comment. Version requirements, hardware requirements, mobility requirements. Either way, outside the scope of this article.

          -matt

  7. Thiago
    December 25th, 2012 at 12:12

    Hi! We can use this procedure with Ubuntu 12.04 if we enable Ubuntu Cloud Archives APT repository. ;-)

    • matthew.mattoon
      December 26th, 2012 at 13:43

      Hi Thiago,

      Thanks for the info.

      All,

      I have not verified Thiago’s input, but on first blush it seems valid. When I have some extra time I will look at writing something up. If you plan to do this please use caution.

      -matt

  8. syno
    December 31st, 2012 at 03:18

    Hey Matthew. Thanks for a great article and the effort you put into this page.

    I have been spending a huge amount of time lately trying to get to know OpenVswitch, but i have not honestly not succeeded that much. I am currently in Kathmandu where i am working witha small WISP and i as of now only have my laptop to run tests etc on. I think its important to be able to demonstate the different ideers and so forth, before implementing them etc. Because of the fact that i am running the most resent libvirt on my ubuntu 12.10 laptop, and the connections available is wireless, making a bridged interface is a bit of a problem.

    Basically i would like to set up a enviroment where i could show them how NIDS works and could benefit there installations etc. If have read that putting up a bridged interface and attaching the vms directly would work as demonstration because they work in promiscuous mode. But again, this seems to not work that well with the connection being all wireless and so forth.
    Lastly but not least, i have thought of using a ubiquiti loco or nanobridge, connecting it to the wired card and so forth…though it seems to be a over the top, amount of work.

    I hope my post is not to confusing, and i hope you will one day get to the point where you will put up a guide on how to set up port mirroring on openVswitch

  9. Shigehiro
    January 8th, 2013 at 10:17

    Hi Matthew

    Thank you for the informative info.
    I could configure OVS on Linux Mint 14 thanks to your great post !
    It works well.

    Shigehiro

  10. Phil
    January 12th, 2013 at 09:59

    Thanks for the info – this was really helpful.

    I found what I think/hope will be a better workaround for the “Waiting for network configuration…” issue at boot up. I just told the networking upstart job to call the openvswitch init scripts before doing its thing by adding them to the pre-start script stanza like this:
    —————
    pre-start script
    mkdir -p /run/network

    #added to get openvswitch to start before networking – gets rid of delay waiting for networking at boot
    /etc/init.d/openvswitch-controller start
    /etc/init.d/openvswitch-switch start

    ifup -a
    end script
    —————–

    This way the switch is already available and “networking” can bring everything up like normal. I did not have to modify the failsafe.conf

  11. Thiago
    January 18th, 2013 at 09:32

    Hi!

    I’m facing this:

    Traceback (most recent call last):
    File “/usr/share/virt-manager/virtManager/asyncjob.py”, line 45, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
    File “/usr/share/virt-manager/virtManager/create.py”, line 1909, in do_install
    guest.start_install(False, meter=meter)
    File “/usr/lib/python2.7/dist-packages/virtinst/Guest.py”, line 1236, in start_install
    noboot)
    File “/usr/lib/python2.7/dist-packages/virtinst/Guest.py”, line 1304, in _create_guest
    dom = self.conn.createLinux(start_xml or final_xml, 0)
    File “/usr/lib/python2.7/dist-packages/libvirt.py”, line 2501, in createLinux
    if ret is None:raise libvirtError(‘virDomainCreateLinux() failed’, conn=self)
    libvirtError: Unable to add bridge ovsbr0p1 port vnet0: Operation not supported

    At virt-manager, I can’t select the listed bridges, they are “unselectable” (gray)..

    Any tips?

    libvirtd –version
    libvirtd (libvirt) 0.9.13

    Thanks!

  12. Thiago
    January 18th, 2013 at 12:21

    BTW, I’m trying this with Ubuntu 12.04 + Ubuntu Cloud Archives, my libvirt is 0.9.13 but my openvswitch is 1.4.0… I’ll try with Ubuntu 12.10 now.
    Tks!

  13. Thiago
    January 21st, 2013 at 08:02

    Well, same problem with Ubuntu 12.10, virt-manager shows: “Unable to complete install: ‘Unable to add bridge ovsbr0 port vnet0: Operation not supported’”
    Are this guide working for the others?

    • matthew.mattoon
      January 21st, 2013 at 09:13

      Hi Thiago,

      Sorry for my delayed response I was out of town last week and unreachable.

      I didn’t test any of my procedures with Virt-Manager. I would suspect that Virt-Manager hasn’t added to support yet for this different network connection. If you use the XML configuration method as described above you should not see any issues correct?

      -matt

  14. Thiago
    January 21st, 2013 at 12:28

    Thank you for your time! No problem!

    But, the error come from libvirt (libvirtError)… Since virt-manager uses libvirt.

    Anyway, I’ll try via CLI in a couple minutes…

    Best,
    Thiago

  15. Thiago
    January 21st, 2013 at 12:33

    BTW, why you install openvswitch-brcompat package? It isn’t used, right? Since libvirt already have native support for Open vSwitch…

  16. Thiago
    January 21st, 2013 at 12:35

    One more… lol

    I’m using:


    auto ovsbr0p1
    iface ovsbr0p1 inet static
    address X.X.X.X
    netmask 24

    Instead of it via DHCP… I think that it is okay (don’t have DHCP within my network)…

  17. Thiago
    January 21st, 2013 at 12:48

    Also, my “ovs-vsctl show” is a bit different, look:

    85e2cda4-d955-4491-b69a-1dacada1301a
    Bridge “ovsbr0″
    Port “eth0″
    Interface “eth0″
    Port “ovsbr0p1″
    Interface “ovsbr0p1″
    type: internal
    Port “ovsbr0″
    Interface “ovsbr0″
    type: internal
    ovs_version: “1.4.3″

    You think that it is okay?

    Thanks!
    Thiago

  18. Thiago
    January 21st, 2013 at 13:09

    Hi!

    I just saw this:

    “The virt-manager doesn’t yet support openvswitch. So there are currently no options available to start a VM using vswitch from virt-manager.”

    Source: http://en.community.dell.com/techcenter/networking/w/wiki/3820.openvswitch-openflow-lets-get-started.aspx

    I’m sorry to flood you blog… Feel free to remove my irrelevant posts…

    Tks!
    Thiago

  19. Thiago
    January 21st, 2013 at 13:27

    YAY!! It is running!! With virt-manager!!

    After creating the VM through virt-manager, even ignoring the error (by manually adding the interface name), I was able to run “virsh edit vmnamehere” and fix the XML file according to your post.

    After that, virt-manager starts the VM normaly using openvswitch anyway…

    I’ll back to Ubuntu 12.04 + Ubuntu Cloud Archives now…

    Thank you!

    Regards,
    Thiago

    • lusheng_jinag
      May 16th, 2013 at 22:44

      hi,hi!
      How can you solve this problem!
      Just fix the XML file?Or whick posts do you remove?

      flys...@hotmail.com
      flyship163

  20. Sascha
    January 28th, 2013 at 10:41

    Thanks for this great article, really helpful.

    Couple of questions to get my head wrapped around this:

    bridge-utils needed a stanza like “bridge_ports” in the interface configuration in /etc/network/interfaces, or you would have to call “brctl addbr xxx” somewhere else in your startup scripts to bring up a bridge during boot. I suppose this is not needed with OVS? If you do a manual “ovs-vsctl add-br” it is boot-persistant? If so, where does it store it’s config?

    The other question I have: Could I keep my eth0 with a static IP and not add it to OVS but instead just create OVS bridge with IP address and then route traffic upstream? This would be helpful in a scenario where you have a hosted server with only one IP address. You could give your guests some RFC1918 addresses and just NAT if traffic needs to go out of the box… basically recreating libvirt’s virbr0 bridge… would that be possible?

    Thanks!

    • matthew.mattoon
      February 5th, 2013 at 18:29

      Hi Sascha,

      When using Openvswitch, you no longer are using bridge-utils (brctl), so you do not need to add this sort of additional entry. You only need to add what is in the article.

      As to your second question a device is either managed by ovs or it is not. So you can choose to not use eth0 with openvswitch, and instead use eth0 as a standard ethernet interface, but then have eth1 be managed by openvswitch.

      To speak to NAT. I don’t believe that is a feature of openvswitch (and I doubt it will be added as that is not a switch function) that said, openvswitch could provision the connections while another device NAT’d the connections, but frankly I wouldn’t focus on that. If you need NAT I wouldn’t go to openvswitch.

      • Sascha
        February 11th, 2013 at 10:32

        Hi Matthew,

        thanks. My question re:NAT was more related to whether OVS needs a physical interface to operate on or whether I could just setup a dummy bridge interface (routing, NAT etc. would be done by the OS). Example (think KVM):

        eth0: 10.10.10.1
        ovsbr0: 192.168.1.1
        –VM 1: 192.168.1.100 (bridged to ovsbr0)
        –VM 2: 192.168.1.101 (bridged to ovsbr0)

        Since ovsbr0 looks like just another interface to Linux, I could use standard utilities (route, iptables) to NAT outgoing traffic from VMs to eth0. e.g. inside VM default route = 192.168.1.1, default route for host = 10.10.10.254, use iptables to NAT 192.168.1.* to host’s IP 10.10.10.1).

        I know I can do this with the default linux bridge and I was wondering if a similar setup is possible with OVS.

        Hope this makes some sense….

        • matthew.mattoon
          February 11th, 2013 at 14:20

          Hi Sascha,

          Thanks for the clarification. I am still not sure of what your premise and ultimate goal here is. Are you asking if you can use OVS inside of a guest? As far as I can see from your notes, you are drawing connections between OVS and legacy bridges where are not there…

          Case in point, when you configure a OVS (openvswitch – notice not a bridge) call it ovsbr0, then you add a couple of VMs VM1 and VM2, each of these are given a port on the openvswitch in the same way that you would connect up a physical nic to a physical switch port. In OVS this is displayed via the naming of the interface. ovsbr0p1, so VM1 would be connected to ovsbr0p1 and VM2 would be connected to ovsbr0p2 and so on. OVS provides a way to perform these physical connections and the switching of the packets. Now that said I see no reason why you could not have port level isolation via VLANs which you could then limit access to via the firewall/router (even if it is on the hypervisor).

          All that to say that I think it is physically and technically possible though I really don’t understand a reasonable use-case for it.

          -matt

          • Sascha
            February 19th, 2013 at 10:50

            Ok, let’s assume the following use case:

            You have one physical server that you rent from a hosting company. You get exactly one IP address. That IP address needs to get assigned to the only physical ethernet port in the machine. Now you want to set up KVM and use OVS instead of the linux default bridge. Since you only have one real IP address (/32) you need to assign RFC1918 addresses to the virtual machines.

            Now if you pull eth0 (the only physical interface) into OVS, ovsbr0 would have the /32 IP address. See where I am getting at? If your ovsbr0 interface has a /32 IP, what are you assigning to the VMs? If you assign RC1918 there is no routing between the VM and the host/outside_net.

            Maybe I am just thinking wrong or I can’t bring it across in english. But my understanding is that in this usecase, you would have to give ovsbr0 an IP address with a large enough subnet mask that allows VMs to be in the same subnet, so that VMs can use the IP of ovsbr0 as their default gateway.

  21. Kiweegie
    February 3rd, 2013 at 09:21

    Hi Matthew, first off thanks for this how-to and the earlier one for Ubuntu 12.04.

    I used your 12.04 instructions successfully in conjunction with this link http://golanzakai.blogspot.co.uk/2012/01/openvswitch-with-virtualbox.html

    My set up is Virtualbox 4.2 on a Linux host with pfsense > openvswitch > multiple vbox clients and this worked perfectly under Ubuntu 12.04 and Mint 13.

    I recently upgraded to Mint 14 so used your new instructions but the clients behind PFsense and Openvswitch can’t get out to the internet – they only get Apipa addresses.

    I am missing some basic understanding of how the host, bridge and switch all bolt together I suspect.

    My ovs-vsctl show output is

    aeaa370c-2a19-4a0a-9f65-e1e8c2a34201
    Bridge “ovsbr0″
    Port “ovsbr0p1″
    Interface “ovsbr0p1″
    type: internal
    Port “ovsbr0″
    Interface “ovsbr0″
    type: internal
    ovs_version: “1.4.3″
    Port “lan0p1″
    Interface “lan0p1″
    …..

    Note there are multiple lan0p* entries I’ve just added the one for brevity.This is all on a laptop with wlan0 and eth0 – I’ve bridged ovsbr0 to eth0 and created the internal port, setup /etc/network/interfaces as required and then added tuntap devices to represent actual ports on the switch.

    I’ve searched virtualbox forums, openvswitch mailing lists and numerous other places but can’t find the answer to what is likely a very simple problem.

    Any pointers gratefully received.

    thanks.

    • matthew.mattoon
      February 5th, 2013 at 18:35

      Hi Klweegle,

      I suspect that perhaps the issue is with using the Ubuntu 12.04 instructions. Mint 14 is closer related to Ubuntu 12.10, and we had to do things differently with regards to the bridges themselves. In 12.10 we don’t do the brcompat stuff, since everything was upgraded enough to avoid it. Which means we don’t have to worry about the brcompat module not loading. That said once you have everything loading properly make sure that you have added the appropriate entries in the /etc/network/interfaces file.

  22. Kiweegie
    February 6th, 2013 at 08:54

    Hi Matthew, thanks for the reply.

    I followed your Ubuntu 12.10 instructions for my Mint 14 laptop, not the 12.04 ones. So I think I’m ok there. I mention the 12.04 instructions as they worked for Mint 13 for me and the steps are not hugely different so struggling to work out where I’m going wrong.

    I omitted installing the openvswitch-brcompat component as 12.10 doesn’t use that (I tried both with and without, made no difference),

    I also omitted the following as I assume they are required for KVM and not Virtualbox (again I’ve tried with and without).

    aptitude apt-show-versions ntp ntpdate vim kvm libvirt-bin vlan virtinst virt-manager virt-viewer openssh-server iperf pv

    Instructions then followed to the letter (very clear and concise they are too) from this point on

    aptitude purge ebtables

    This includes amending the /etc/network/interfaces file to include

    # The primary network interface
    auto eth0
    iface eth0 inet manual
    up ifconfig $IFACE 0.0.0.0 up
    down ifconfig $IFACE down

    # hypervisor interface
    auto ovsbr0p1
    iface ovsbr0p1 inet dhcp

    I “think” the openvswitch base install is ok and functioning its how to add ports to the switch (use tuntap devices per the other link in my earlier post or ovs-vsctl add-port command?) and then tie them to virtualbox guests.

    Appreciate your instructions are KVM specific but hope you or another person finding this page can shed some light on where I’m going wrong.

    Thanks again for this how to and the time you have taken to reply, it’s appreciated.

  23. Kiweegie
    February 6th, 2013 at 16:56

    Hi Matthew,

    I did some more digging on this tonight and it’s now working. Suspect working all along however a network config issue on the Pfsense LAN side (from a snapshot on earlier working install) was preventing DHCP delivery to the guests behind pfsense.

    To recap for anyone else viewing these pages I have used Matthews excellent tutorial here on a Linux Mint 14 host running Virtualbox with Pfsense as DHCP/router/firewall – openvswitch – other VBox clients.

    PFsense has 2 ports in Virtualbox, one bridged to wlan0 (WAN) and the other to lan0p1 (LAN)

    As I’m using Virtualbox not KVM I omitted the following installation steps in Matthews tutorial

    # apt-get install aptitude apt-show-versions ntp ntpdate vim kvm libvirt-bin vlan virtinst virt-manager virt-viewer openssh-server iperf pv openvswitch-brcompat

    # virsh net-destroy default
    # virsh net-autostart –disable default

    Stop Libvirt and Qemu
    This will prevent libvirt from bringing up the legacy bridge
    # service libvirt-bin stop
    # service qemu-kvm stop

    The above prove to be unnecessary if using Virtualbox and not KVM

    You will also not need

    # apt-get purge ebtables

    running this command gihves response ebtables not installed

    Otherwise follow Mattews steps as he lays out and you’ll be fine.

    On top of this I referred to this link which details how to create tuntap devices to act as ports on the switch.

    http://golanzakai.blogspot.co.uk/2012/01/openvswitch-with-virtualbox.html

    These are not persistent on reboot so you’re best to add to a script. Here’s mine which deletes any existing tuntap devices (lan0p1-lan0p16) then creates them again and binds them to ovsbr0

    # vi vboxstart.sh

    Add the following to the script and save

    # delete existing lan0p interfaces
    for tap in `seq 1 16`; do tunctl -d lan0p$tap; done;

    # delete existing lan0p ports attached to ovsbr0
    for tap in `seq 1 16`; do ovs-vsctl del-port ovsbr0 lan0p$tap; done;

    # create tap devices on the host machine so we can bind them to the guests
    for tap in `seq 1 16`; do tunctl -u tom -t lan0p$tap; done;

    # Bring the tap interfaces up:
    for tap in `seq 1 16`; do ip link set lan0p$tap up; done;

    # Now we use ovs-vsctl to bind the tap devices to ovsbr0 switch
    for tap in `seq 1 16`; do ovs-vsctl add-port ovsbr0 lan0p$tap; done;

    Now edit file so it’s executable

    # chmod +x vboxstart.sh

    Hope this saves someone else some time!

  24. Calreth
    February 19th, 2013 at 14:51

    Following the same configuration, I’m getting the following error when I try to start my vm

    error: Unable to add bridge ovsbr0 port vnet0: Operation not supported

    Any ideas?

  25. matthew.mattoon
    February 19th, 2013 at 16:51

    Sascha :

    Ok, let’s assume the following use case:

    You have one physical server that you rent from a hosting company. You get exactly one IP address. That IP address needs to get assigned to the only physical ethernet port in the machine. Now you want to set up KVM and use OVS instead of the linux default bridge. Since you only have one real IP address (/32) you need to assign RFC1918 addresses to the virtual machines.

    Now if you pull eth0 (the only physical interface) into OVS, ovsbr0 would have the /32 IP address. See where I am getting at? If your ovsbr0 interface has a /32 IP, what are you assigning to the VMs? If you assign RC1918 there is no routing between the VM and the host/outside_net.

    Maybe I am just thinking wrong or I can’t bring it across in english. But my understanding is that in this usecase, you would have to give ovsbr0 an IP address with a large enough subnet mask that allows VMs to be in the same subnet, so that VMs can use the IP of ovsbr0 as their default gateway.

    Basically what you are trying to accomplish really has very little to do with OVS. You are looking to multihome your machine, though this can be accomplished with a single eth adapter (IP aliasing, VLAN, et al). Your VMs are configured to use private address space, and communicate with their gateway which is also on private address space (ip alias) the router needs to then take over and route that outbound traffic out (and of course NAT it) but my point is that this doesn’t have anything to do with OVS. OVS handles the virtual “physical” connectivity. Something else needs to do the routing and NAT in order for something like this to work.

    -matt

  26. matthew.mattoon
    February 19th, 2013 at 17:10

    Calreth :

    Following the same configuration, I’m getting the following error when I try to start my vm

    error: Unable to add bridge ovsbr0 port vnet0: Operation not supported

    Any ideas?

    Shot in the dark, because you didn’t give me a lot of context. Looks like you are trying to use virt-manager instead of libvirt directly (as I did in my article). If this is the case refer to comment #26. Basically you need to fix your XML manually and then start your VMs however you want, virt-manager is trying to connect to a legacy bridge, which you don’t have by following this article.

    -matt

    • Calreth
      February 19th, 2013 at 20:23

      I’m running it through virsh. That is using libvirt directly, no?

      I did change the XML to point the source bridge to the one I created in ovs-vsctl, as well as specifying the port type as openvswitch.

      Is there a way to dump a trace of exactly what happened (is it somewhere in the libvirt logs, perhaps?) that could further help me understand the problem? I’m rather stumped.

      There is no need for the bridge compatibility module, right? I’ve also tried turning that on but I seem to have a separate issue with the brcomp module not loading.

      • matthew.mattoon
        February 19th, 2013 at 20:27

        Virsh is a utility provided by libvirt, so yes your assumption is correct. Can you paste your XML? I suspect that is the key. Also for clarity please post your OS, and the ovs-vsctl show

        • Calreth
          February 19th, 2013 at 21:31

          I’m running Ubuntu 12.04, with libvirt upgraded to 1.0.2 for the native OVS support.

          # Versions
          ovs-vsctl (Open vSwitch) 1.4.0+build0
          Compiled Feb 18 2013 13:13:22

          libvirtd –version
          libvirtd (libvirt) 1.0.2

          # vsctl show – I don’t have the physical eth interface bridged because I don’t want my client OS to have network access to the physical network at this point
          012603db-d204-4381-8cae-50e3f15c35cc
          Bridge “br0″
          Port “br0″
          Interface “br0″
          type: internal
          Port “br0p1″
          Interface “br0p1″
          type: internal
          ovs_version: “1.4.0+build0″

          # XML config – I’ve snippet the other stuff for brevity.

        • Calreth
          February 19th, 2013 at 21:34

          I think the XML config got snipped off or filtered out because it might been treated as tags, but I’ve it here > http://pastebin.com/0EpcGrJC

          • matthew.mattoon
            February 20th, 2013 at 08:09

            Your XML is not complete, looks like libvirt is not adding the target dev and other port specific information. I would recommend following the article completely. And once you get it working as I had it working, then backtrack and undo the pieces you don’t want (external connectivity etc).

  27. Calreth
    February 21st, 2013 at 00:24

    Thanks, will try to see if I can do a clean install and reproduce everything exactly.

    If I don’t require the external connectivity, I wouldn’t need to add the physical ethernet device to the bridge, right? What about the internal ovsbr0p1 port?

    My final ideal configuration would be to have a GRE tunnel to another open vswitch host, and that the VMs on each of them be allowed to communicate with each other only, but not any other networks.

    matthew.mattoon :Your XML is not complete, looks like libvirt is not adding the target dev and other port specific information. I would recommend following the article completely. And once you get it working as I had it working, then backtrack and undo the pieces you don’t want (external connectivity etc).

  28. F6
    April 30th, 2013 at 00:26

    In ur article u have wrote, to reboot the system. Correct me, if u reboot the system the ovs process will get killed which means everything u configured will be lost. The changes in /etc/network/interfaces is bound to give u nightmare after the reboot. I am not sure but this is b’coz the during reboot system will not get “ovsbr0p1″ device. (I have tried this on Ubuntu 12.10, it doesn’t work. My ovsdb-server is running on tcp port instead of by default unix socket). Will wait for ur reply.

Comments are closed.