Bandwidth rate shaping with tc stops working after upgrade to SLES 10 SP2

SUSE Linux Enterprise Server 10 Service Pack 2
SUSE Linux Enterprise Server 10

Bandwidth rate shaping with the Linux kernel Traffic Control utility (/usr/sbin/tc) stops working after upgrading from SUSE Linux Enterprise Server (SLES) 10 and earlier versions to SLES 10 Service Pack (SP) 2.

For example a policy to limit the outgoing bandwidth for eth0 to approximately 430 KB/s (kilobytes) could be implemented like this :

# tc qdisc add dev eth0 root handle 1:0 cbq bandwidth 1000Mbit avpkt 1000 cell 8
# tc class add dev eth0 parent 1:0 classid 1:1 cbq bandwidth 1000Mbit rate 3482kbit prio 1 avpkt 1000 bounded isolated
# tc filter add dev eth0 parent 1:0 protocol ip u32 match ip src 192.168.3.4 flowid 1:1

The reason for this that TCP Segmentation Offload (TSO) has been enabled by default in the newer network cards. And while SLES 9 and earlier SLES 10 kernels did disable this feature by default (since it did not work on most hardware), this setting is now left enabled in the newer kernels.

TCP segmentation offload is used to reduce the CPU overhead of TCP/IP on fast networks. TSO breaks down large groups of data sent over a network into smaller segments that pass through all the network elements between the source and destination. This type of offload relies on the network interface controller (NIC) to segment the data and then add the TCP, IP and data link layer protocol headers to each segment. The NIC must support TSO, which is also called large segment offload (LSO).

Since TSO enables transmission of data "outside" the linux kernel to optimize the throughput without bugging the CPU, it then breaks the traffic control shaping because the kernel only sees the "tips of the icebergs" going through the NIC.

To identify whether TSO is enabled on a NIC, use a command like

# ethtool -k eth0

This should produce output like the following :
Offload parameters for eth0:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: on

The solution to the problem is to disable TSO for the interface, which can be done by following these steps as a test :

1. Remove any active tc settings from the interface with a command like

   # tc qdisc del root dev eth0
   

2. Change the TSO setting to off for the interface. If it is eth0, the command would be

   # ethtool -K eth0 tso off
   

   (note the capital K)
3. Verify that it has been disabled:

   # ethtool -k eth0
   

4. Reapply the tc settings - if using the example above :

   # tc qdisc add dev eth0 root handle 1:0 cbq bandwidth 1000Mbit avpkt 1000 cell 8
   # tc class add dev eth0 parent 1:0 classid 1:1 cbq bandwidth 1000Mbit rate 3482kbit prio 1 avpkt 1000 bounded isolated
   # tc filter add dev eth0 parent 1:0 protocol ip u32 match ip src 192.168.3.4 flowid 1:1
   

Now verify that the bandwidth gets limited as expected - e.g. by copying a large file with scp. Note that it takes a little while before the rate settles around the rate that has been defined.
If it works, then the next step is to implement a permanent solution which disables TSO for the network card(s) in question during system boot :

1. Open the /etc/sysconfig/network/ifcfg- file that coresponds with the card in an editor
2. Add the following setting to the ETHTOOL_OPTIONS parameter :
   -K iface tso off 
   so that the line e.g. reads :
   ETHTOOL_OPTIONS='-K iface tso off'
3. Save the file
4. Repeat steps 1-3 for each card that should have TSO disabled
5. Reboot the system to implement the new settings
6. Verify that TSO has been disabled for each interface with 

   # ethtool -k ethX