Hassle-free Xen Networking
This document (7000616) is provided subject to the disclaimer at the end of this document.
Environment
SUSE Linux Enterprise Server 10 Service Pack 2
Situation
Many problems arise from the use of the network-bridge script from SLES 10 SP2.
For example the following might occur:
For example the following might occur:
- The network-bridge script does not work for multiple xen bridges
- The network-bridge script does not work very well with dhcp enabled devices
- The network-bridge script does not transfer additional routes to xen bridge
- You want to get rid of the "useless" P-devices and have a consistent network setup even when booting between normal and xen kernel.
Resolution
We can't really provide a single step by step solution. Instead we want to describe how it works.
First we recommend to boot into the normal (not xen) kernel and configure all network devices as you like to have a working setup.
Now for each xenbridge you later want to use create a configuration file for a normal linux bridge device that can be handled well by ifup/ifdown scripts. Linux bridges are identified by the ifcfg filename
ifcfg-br
The following things are important:
IE:
As final step (not necessary if you only use one bridge and never configured xen to use a bridge by name) you may need to adapt any existing guest configuration to use the new bridge, instead ofxenbr. Open each guest configuration file (they are located in directory /etc/xen/vm/ and search for a line similar to
vif=[ 'mac=00:16:3e:0a:15:6c,bridge=xenbr0', ]
vif=[ 'mac=00:16:3e:7e:f3:c7,model=ne2k_pci,type=ioemu', ]
vif=[ 'mac=00:16:3e:7c:6b:f3', ]
and replace it with:
vif=[ 'mac=00:16:3e:0a:15:6c,bridge=br0', ]
vif=[ 'mac=00:16:3e:7e:f3:c7,model=ne2k_pci,type=ioemu,bridge=br0', ]
vif=[ 'mac=00:16:3e:7c:6b:f3,bridge=br0', ]
This was the last step. Now you can boot again into the xen kernel and everything should work as expected.
Note: If you want to create a new guest via virt-manager later, make sure you edit the networking adapter option and replace the source from default to your bridge (like br0). This is only needed if you have more than one bridge.
Note: If you are using the SuSEFirewall2 to secure your Xen Host and Guests you may also need to adapt the /etc/sysconfig/SuSEfirewall2 to use your bridge (like br0) instead of xenbr0.
Tip: If you like to use YaST2 to configure the networking and you use a bonding device you have to set the Device Activation to on boot time and assign a static ip address 0.0.0.0 and Subnet Mask 255.255.255.255 for the bonding device, else the configuration dialog for the bridge does not show up any selectable interface.
Update 07.Feb.2009:
When using arp-monitoring for bonding it is essential that the bonding device has an ip address assigned. When the bonding device has no ip address you will notice slave flapping in the logfiles. To not confuse the routing it is also needed that no route supersedes the route from the bridge device. To assign a ip address to the bonding device please use following solution.
Create a script cp_ip_br_to_bond in /etc/sysconfig/network/scripts with following content:
------cp_ip_br_to_bond--------------
#!/bin/bash
# get bonding device
. /etc/sysconfig/network/ifcfg-$1
# cp ip from bridge to bonding device
ip addr show dev $2 | egrep '^ *inet ' | sed -e "
s/inet/ip addr add/
s@\([0-9]\+\.[0-9]\+\.[0-9]\+\.[0-9]\+/[0-9]\+\)@\1@
s/$2/dev ${BRIDGE_PORTS} label ${BRIDGE_PORTS}/
s/secondary//
" | sh -e
# Remove automatic routes on bonding device
ip route list | sed -ne "
/dev ${BRIDGE_PORTS}\( \|$\)/ {
s/^/ip route del /
p
}" | sh -e
--------------------------------------------------
make the script executable with
chmod +x /etc/sysconfig/network/scripts/cp_ip_br_to_bond
and include it in the ifcfg-br0 file as
POST_UP_SCRIPT='cp_ip_br_to_bond'
We are still working on a final fix inside the ifup/ifdown scripts without the need of this workaround. We will post an update as soon we have released a final fix.
Update 01.Aug.2008:
- fixed device order in static routing example
- you might also want to set BRIDGE_FORWARDDELAY="0" in your /etc/sysconfig/ifcfg-br configuration file. This disables the default forwarding delay (15 sec.) and the interface is faster ready to send / receive.
First we recommend to boot into the normal (not xen) kernel and configure all network devices as you like to have a working setup.
Now for each xenbridge you later want to use create a configuration file for a normal linux bridge device that can be handled well by ifup/ifdown scripts. Linux bridges are identified by the ifcfg filename
ifcfg-br
The following things are important:
- insert your original device name to the option BRIDGE_PORTS of the bridge configuration file
- transfer the ip configuration from the original device to the bridge device (BOOTPROTO,IPADDR,NETMASK etc.)
- in the original device configuration set the BOOTPROTO to "none"
Here is an example:
The original network device is bond0 and we want to create the xenbridge as br0.
---original "/etc/sysconfig/network/ifcfg-bond0" --
BONDING_MASTER='yes'
BONDING_MODULE_OPTS='mode=balance-rr miimon=100'
BONDING_SLAVE0='bus-pci-0000:00:07.0'
BONDING_SLAVE1='bus-pci-0000:00:07.1'
BOOTPROTO='dhcp'
BROADCAST=''
ETHTOOL_OPTIONS=''
IPADDR=''
MTU=''
NAME=''
NETMASK=''
NETWORK=''
REMOTE_IPADDR=''
STARTMODE='auto'
USERCONTROL='no'
----------------------------------------------------
---adapted "/etc/sysconfig/network/ifcfg-bond0" --
BONDING_MASTER='yes'
BONDING_MODULE_OPTS='mode=balance-rr miimon=100'
BONDING_SLAVE0='bus-pci-0000:00:07.0'
BONDING_SLAVE1='bus-pci-0000:00:07.1'
BOOTPROTO='none'
BROADCAST=''
ETHTOOL_OPTIONS=''
IPADDR=''
MTU=''
NAME=''
NETMASK=''
NETWORK=''
REMOTE_IPADDR=''
STARTMODE='auto'
USERCONTROL='no'
---------------------------------------------------
----new "/etc/sysconfig/network/ifcfg-br0"---------
BOOTPROTO='dhcp'
BRIDGE='yes'
BRIDGE_PORTS='bond0'
BROADCAST=''
ETHTOOL_OPTIONS=''
IPADDR=''
MTU=''
NAME=''
NETMASK=''
NETWORK=''
REMOTE_IPADDR=''
STARTMODE='auto'
USERCONTROL='no'
---------------------------------------------------
If you have static IP addresses with static routing configuration, you might also want to adjust the file /etc/sysconfig/networks/routes and insert the bridgename instead of the original device or the - wild-card for any device. (This prevents warnings about the routing configuration during network start)
before:
192.168.23.0 192.168.42.23 255.255.255.0 - bond0
default 192.168.42.23 - -
after:
192.168.23.0 192.168.42.23 255.255.255.0 - br0
default 192.168.42.23 - br0
That is all you need to do, now your network will work in the normal and the xen environment. The next step is to disable the network-bridge script. To do this, open the file /etc/xen/xend-config.sxp into your favorite editor, search for any line similar to
(network-script network-bridge)
and either leave it empty or set it to use /bin/true (as the default, when commented out, is to use the network bridge):
The original network device is bond0 and we want to create the xenbridge as br0.
---original "/etc/sysconfig/network/ifcfg-bond0" --
BONDING_MASTER='yes'
BONDING_MODULE_OPTS='mode=balance-rr miimon=100'
BONDING_SLAVE0='bus-pci-0000:00:07.0'
BONDING_SLAVE1='bus-pci-0000:00:07.1'
BOOTPROTO='dhcp'
BROADCAST=''
ETHTOOL_OPTIONS=''
IPADDR=''
MTU=''
NAME=''
NETMASK=''
NETWORK=''
REMOTE_IPADDR=''
STARTMODE='auto'
USERCONTROL='no'
----------------------------------------------------
---adapted "/etc/sysconfig/network/ifcfg-bond0" --
BONDING_MASTER='yes'
BONDING_MODULE_OPTS='mode=balance-rr miimon=100'
BONDING_SLAVE0='bus-pci-0000:00:07.0'
BONDING_SLAVE1='bus-pci-0000:00:07.1'
BOOTPROTO='none'
BROADCAST=''
ETHTOOL_OPTIONS=''
IPADDR=''
MTU=''
NAME=''
NETMASK=''
NETWORK=''
REMOTE_IPADDR=''
STARTMODE='auto'
USERCONTROL='no'
---------------------------------------------------
----new "/etc/sysconfig/network/ifcfg-br0"---------
BOOTPROTO='dhcp'
BRIDGE='yes'
BRIDGE_PORTS='bond0'
BROADCAST=''
ETHTOOL_OPTIONS=''
IPADDR=''
MTU=''
NAME=''
NETMASK=''
NETWORK=''
REMOTE_IPADDR=''
STARTMODE='auto'
USERCONTROL='no'
---------------------------------------------------
If you have static IP addresses with static routing configuration, you might also want to adjust the file /etc/sysconfig/networks/routes and insert the bridgename instead of the original device or the - wild-card for any device. (This prevents warnings about the routing configuration during network start)
before:
192.168.23.0 192.168.42.23 255.255.255.0 - bond0
default 192.168.42.23 - -
after:
192.168.23.0 192.168.42.23 255.255.255.0 - br0
default 192.168.42.23 - br0
That is all you need to do, now your network will work in the normal and the xen environment. The next step is to disable the network-bridge script. To do this, open the file /etc/xen/xend-config.sxp into your favorite editor, search for any line similar to
(network-script network-bridge)
and either leave it empty or set it to use /bin/true (as the default, when commented out, is to use the network bridge):
IE:
network-script
or
network-script /bin/true
As final step (not necessary if you only use one bridge and never configured xen to use a bridge by name) you may need to adapt any existing guest configuration to use the new bridge, instead ofxenbr. Open each guest configuration file (they are located in directory /etc/xen/vm/ and search for a line similar to
vif=[ 'mac=00:16:3e:0a:15:6c,bridge=xenbr0', ]
vif=[ 'mac=00:16:3e:7e:f3:c7,model=ne2k_pci,type=ioemu', ]
vif=[ 'mac=00:16:3e:7c:6b:f3', ]
and replace it with:
vif=[ 'mac=00:16:3e:0a:15:6c,bridge=br0', ]
vif=[ 'mac=00:16:3e:7e:f3:c7,model=ne2k_pci,type=ioemu,bridge=br0', ]
vif=[ 'mac=00:16:3e:7c:6b:f3,bridge=br0', ]
This was the last step. Now you can boot again into the xen kernel and everything should work as expected.
Note: If you want to create a new guest via virt-manager later, make sure you edit the networking adapter option and replace the source from default to your bridge (like br0). This is only needed if you have more than one bridge.
Note: If you are using the SuSEFirewall2 to secure your Xen Host and Guests you may also need to adapt the /etc/sysconfig/SuSEfirewall2 to use your bridge (like br0) instead of xenbr0.
Tip: If you like to use YaST2 to configure the networking and you use a bonding device you have to set the Device Activation to on boot time and assign a static ip address 0.0.0.0 and Subnet Mask 255.255.255.255 for the bonding device, else the configuration dialog for the bridge does not show up any selectable interface.
Update 07.Feb.2009:
When using arp-monitoring for bonding it is essential that the bonding device has an ip address assigned. When the bonding device has no ip address you will notice slave flapping in the logfiles. To not confuse the routing it is also needed that no route supersedes the route from the bridge device. To assign a ip address to the bonding device please use following solution.
Create a script cp_ip_br_to_bond in /etc/sysconfig/network/scripts with following content:
------cp_ip_br_to_bond--------------
#!/bin/bash
# get bonding device
. /etc/sysconfig/network/ifcfg-$1
# cp ip from bridge to bonding device
ip addr show dev $2 | egrep '^ *inet ' | sed -e "
s/inet/ip addr add/
s@\([0-9]\+\.[0-9]\+\.[0-9]\+\.[0-9]\+/[0-9]\+\)@\1@
s/$2/dev ${BRIDGE_PORTS} label ${BRIDGE_PORTS}/
s/secondary//
" | sh -e
# Remove automatic routes on bonding device
ip route list | sed -ne "
/dev ${BRIDGE_PORTS}\( \|$\)/ {
s/^/ip route del /
p
}" | sh -e
--------------------------------------------------
make the script executable with
chmod +x /etc/sysconfig/network/scripts/cp_ip_br_to_bond
and include it in the ifcfg-br0 file as
POST_UP_SCRIPT='cp_ip_br_to_bond'
We are still working on a final fix inside the ifup/ifdown scripts without the need of this workaround. We will post an update as soon we have released a final fix.
Update 01.Aug.2008:
- fixed device order in static routing example
- you might also want to set BRIDGE_FORWARDDELAY="0" in your /etc/sysconfig/ifcfg-br configuration file. This disables the default forwarding delay (15 sec.) and the interface is faster ready to send / receive.
Additional Information
Disclaimer
This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.
- Document ID:7000616
- Creation Date: 06-Jun-2008
- Modified Date:03-Mar-2020
-
- SUSE Linux Enterprise Server
For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback[at]suse.com
