Sunday, July 12, 2015

QEMU/KVM Bridged Network with TAP interfaces

In my previous post, Rapid Linux Kernel Dev/Test with QEMU, KVM and Dracut, I described how build and boot a Linux kernel quickly, making use of port forwarding between hypervisor and guest VM for virtual network traffic.

This post describes how to plumb the Linux VM directly into a hypervisor network, through the use of a bridge.

Start by creating a bridge on the hypervisor system:
> sudo ip link add br0 type bridge

Clear the IP address on the network interface that you'll be bridging (e.g. eth0).
Note: This will disable network traffic on eth0!
> sudo ip addr flush dev eth0
Add the interface to the bridge:
> sudo ip link set eth0 master br0


Next up, create a TAP interface:
> sudo ip tuntap add dev tap0 mode tap user $(whoami)
The user parameter ensures that the current user will be able to connect to the TAP interface.

Add the TAP interface to the bridge:
> sudo ip link set tap0 master br0

Make sure everything is up:
> sudo ip link set dev br0 up
> sudo ip link set dev tap0 up

The TAP interface is now ready for use. Assuming that a DHCP server is available on the bridged network, the VM can now obtain an IP address during boot via:
> qemu-kvm -kernel arch/x86/boot/bzImage \
           -initrd initramfs \
           -device e1000,netdev=network0,mac=52:55:00:d1:55:01 \
           -netdev tap,id=network0,ifname=tap0,script=no,downscript=no \
           -append "ip=dhcp rd.shell=1 console=ttyS0" -nographic

The MAC address is explicitly specified, so care should be taken to ensure its uniqueness.

The DHCP server response details are printed alongside network interface configuration. E.g.
[    3.792570] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
[    3.796085] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[    3.812083] Sending DHCP requests ., OK
[    4.824174] IP-Config: Got DHCP answer from 10.155.0.42, my address is 10.155.0.1
[    4.825119] IP-Config: Complete:
[    4.825476]      device=eth0, hwaddr=52:55:00:d1:55:01, ipaddr=10.155.0.1, mask=255.255.0.0, gw=10.155.0.254
[    4.826546]      host=rocksolid-sles, domain=suse.de, nis-domain=suse.de
...

Didn't get an IP address? There are a few things to check:
  • Confirm that the kernel is built with boot-time DHCP client (CONFIG_IP_PNP_DHCP=y) and E1000 network driver (CONFIG_E1000=y) support.
  • Check the -device and -netdev arguments specify a valid e1000 TAP interface.
  • Ensure that ip=dhcp is provided as a kernel boot parameter, and that the DHCP server is up and running.
Happy hacking

 Update 20161223:
  • Use 'ip' instead of 'brctl' to manipulate the bridge device - thanks Yagamy Light!
Update 20171220:
  • Use 'ip tuntap' instead of 'tunctl' to create the TAP interface - thanks Johannes!
Update 20200930:
  • For performance reasons, I strongly recommend using virtio network adapters instead of e1000.

18 comments:

  1. I was looking for solid, simple examples for how to bridge a QEMU guest to my host without using SLIRP. This is the most distro agnostic and to the point example I've found so far, thank you for putting this together!

    ReplyDelete
  2. Glad to hear it was helpful. Thanks for the feedback Kenneth!

    ReplyDelete
  3. Thank you for simple instructions you put together. It has been great help to us.

    ReplyDelete
  4. Great tutorial, will be using this to create VM to VM taps for SDN services.

    ReplyDelete
  5. Thank you very .. it was very helpful for me

    ReplyDelete
  6. An important note for static ip addresses: the ip address that was previously at eth0, now have to be set to `br0` device, then LAN starts working.

    I also think `sudo /sbin/brctl addbr br0` can be changed to `sudo ip link add br0 type bridge`, and `sudo /sbin/brctl addif br0 eth0` → `sudo ip link set eth0 master br0`.

    ReplyDelete
    Replies
    1. Thanks for the brctl -> ip link type bridge recommendation. I'll give it a test and update the tutorial

      Delete
  7. Thanks for the short and informative tutorial. I have followed this and it worked. I can ping guests from host. From guests I can ping host, my internet router and internet hosts. However I can't ping guest from guest. Would you update your article on how to accomplish this?

    ReplyDelete
    Replies
    1. Hi Dimon, guest to guest communication works fine for me. Could your firewall be getting in the way?

      Delete
  8. Chester T. Field Esq.December 19, 2016 at 6:47 AM

    Thanks for posting this, I had a meatball of a time getting qemu to play nice with my local network.

    ReplyDelete
  9. Hey, in the article you mentioned to ensure that the dhcp server is up and running. By that did you mean I should start a dhcp server on the host itself? If so, could you tell me as to what it's configuration should be?

    ReplyDelete
  10. Hi Prashast, if you're bridging your VM with a physical network device (e.g. eth0 in this example) then a DHCP server will very likely already be running on that network.

    If not, then you could start your own on br0 via:
    > dnsmasq --no-hosts --no-resolv --interface="br0" --dhcp-range="192.168.31.10,192.168.31.20,12h"

    ReplyDelete
  11. I always get confused by taps. This is by far the best tutorial on bridging the tap. I like that you didn't use brctl but ip link instead. It just misses a line about assinging IP to bridge. Usually it would be like sudo dhclient br0.

    ReplyDelete
    Replies
    1. Thanks for the feedback. I deliberately avoided bridge IP assignment, as I wanted to keep the documentation as simple as possible. I'll consider adding it in future if there's enough demand.

      Delete
  12. Thank you for the nice write up. What if the guest VM should be in a different subnet than the host?

    sudo ip link add br0 type bridge
    sudo ip tuntap add dev tap0 mode tap user $(whoami)
    sudo ip link set tap0 master br0
    sudo ip link set dev br0 up
    sudo ip link set dev tap0 up
    sudo sysctl -w net.ipv4.ip_forward=1
    sudo ip route add to $prefix dev tap0

    sudo qemu-system-x86_64 -enable-kvm -hda ./tpot/vm.img -device e1000,netdev=network0,mac=52:54:00:09:9b:15 -netdev tap,id=network0,ifname=tap0,script=no,downscript=no -display none -daemonize

    and then static network configuration in /etc/network/interfaces in the guest did not work for me. I cant ping from host-->guest. :(

    Cheers. MN

    ReplyDelete
  13. Thank you so much for a great guide. Would you please post
    1. static IP instructions (preferably the Arch linux way)
    2. solution on how to make this changes permanent so that they survive the reboot. Your guide works for me but resets after reboot. I have to go through the whole thing again with every reboot.
    Thanks!

    ReplyDelete
  14. I had to also run dhclient on br0 afterwards on the host, F29.

    ReplyDelete
  15. Sweet and simple - and it works. Excellent writeup. I used it for my Manjaro setup.

    ReplyDelete

Comments are moderated due to spammer abuse.