PerlStalker’s SysAdmin Notes

Notes from the life of a systems administrator

Life With KVM: Networking

Originally posted at [2008-10-20 Mon 20:46]

I spent a couple of days a while back trying to figure out why I was seeing bizarre bridge and network errors in my KVM host’s syslog and a VM that only two of three NICs worked at a time. Turns out that there is a very simple fix for both problems.

First, let’s start with the KVM host network configuration. Here’s the basic config for a host with two NICs. It gives you a pretty basic setup for your KVM host network on Debian systems. You should be able to adapt them for other systems. The important thing to note is that you will need a bridge device if you want your VMs to be directly accessible.

# The primary network interface
auto eth0
iface eth0 inet manual

auto br0
iface br0 inet static
      bridge_ports eth0
      bridge_hello 2
      bridge_maxage  12
      bridge_stp on
      bridge_maxwait 5
      address 10.10.134.89
      netmask 255.255.255.192
      network 10.10.134.64
      broadcast 10.10.134.127
      gateway 10.10.134.126

# Storage network
auto eth1
iface eth1 inet static
      address 0.0.0.0
      netmask 0.0.0.0
      mtu 9000

auto br1
iface br1 inet static
      bridge_ports eth1
      bridge_hello 2
      bridge_maxage  12
      bridge_stp on
      bridge_maxwait 5
      address 192.168.0.11
      netmask 255.255.255.0

This setup gives me a pair of bridged interfaces I can attach my guests to. The first is an Internet facing network (address changed to protect the guilty). The second is a storage network that allows my guests to mount iSCSI targets directly. (I have more interfaces on other networks but this will do for this discussion.)

A couple of things of note. I have turned up the MTU on eth1 to 9000. This network uses jumbo frames to provide a little bit of a boost to the iSCSI links. It’s certainly, not required.

Update (2008-10-23 20:07): I was reading a little more today and came across a brief discussion of jumbo frames on qemu and, by extension, kvm. There seems to be a 4K limit in qemu’s VLAN implementation. Virtio-net seems to work fine with jumbo frames.

The other thing is that I have turned STP on. I had problems with live migration with STP off. Turning it on fixed my live migration but contributed to my network issues. Let me explain.

A healthy bridge with STP will look something like this:

$ brctl showstp br1
br1
 bridge id              8000.001ec9cbeb53
 designated root        8000.0019b9cdb57d
 root port                 1                    path cost                  4
 max age                  12.00                 bridge max age            12.00
 hello time                2.00                 bridge hello time          2.00
 forward delay            15.00                 bridge forward delay      15.00
 ageing time             300.00
 hello timer               0.00                 tcn timer                  0.00
 topology change timer     0.00                 gc timer                   9.38
 flags

eth1 (1)
 port id                8001                    state                forwarding
 designated root        8000.0019b9cdb57d       path cost                  4
 designated bridge      8000.0019b9cdb57d       message age timer         10.43
 designated port        8001                    forward delay timer        0.00
 designated cost           0                    hold timer                 0.00
 flags

tap10 (2)
 port id                8002                    state                forwarding
 designated root        8000.0019b9cdb57d       path cost                100
 designated bridge      8000.001ec9cbeb53       message age timer          0.00
 designated port        8002                    forward delay timer        0.00
 designated cost           4                    hold timer                 0.00
 flags

Notice how all of the interfaces in the bridge are in the forwarding state. What happened to me was that I had certain VM tap devices get put into a blocking state. What that happens no traffic is sent through the interface.

What took me all those days to find out was why the interface was stuck blocking. Basically, qemu (which kvm uses) has the notion of virtual networks. The docs call them vlans but they are not the same as 802.11q VLANs. However, they serve the same purpose. NICs in different qemu-vlans are in separate broadcast domains.

So, what does that mean? When I set the VM up the first time, I had this for my network config.

... -net nic,model=virtio -net tap,script=/etc/kvmctl/scripts/kvm-ifup-test\
 -net nic,model=virtio -net tap,script=/etc/kvmctl/scripts/kvm-ifup-storage

Each net option has a parameter, vlan, that if left unset (as above) it defaults to 0. In this case, I had both NICs in the same broadcast domain (qemu-vlan 0). When STP tried to figure out where the taps were, it got confused because it saw traffic from one NIC on the other. Basically, it did what it’s supposed to do and shut down one of them to prevent an ethernet loop.

The fix is to set the vlan on each virtual NIC, like so:

... -net nic,vlan=0,model=virtio -net tap,script=/etc/kvmctl/scripts/kvm-ifup-test,vlan=0\
 -net nic,vlan=1,model=virtio -net tap,script=/etc/kvmctl/scripts/kvm-ifup-storage,vlan=1

No more problems. :-) I suggest that you use the same vlan number for each network. I use the number on the bridge.

For the curious, the kvm-ifup-* scripts look like this. (This is kvm-ifup-test .)

#!/bin/bash

CONFDIR=/etc/kvmctl

. $CONFDIR/scripts/kvm-ifup.conf

$IFCONFIG $1 0.0.0.0 up
$BRCTL addif $IF_TEST $1

And kvm-ifup.conf

IFCONFIG=/sbin/ifconfig
BRCTL=/usr/sbin/brctl

# Interface defaults
IF_DMZ=br3
IF_TEST=br0
IF_MS=br4
IF_STORAGE=br1
IF_BACKUP=br2

# Load host specific overrides
UNAME=`uname -n`
if [ -r $CONFDIR/scripts/kvm-ifup-${UNAME}.conf ]; then
   . $CONFDIR/scripts/kvm-ifup-${UNAME}.conf
fi

For further details on qemu network settings, see QEMU Network Emulation.

Comments