Applying SLES 11 SP 1 Causing Communication Issues

This document (7007649) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 11 Service Pack 2

SUSE Linux Enterprise Server 11 Service Pack 1

Situation

After patching to SP 1 on SLES 11 administrators are noticing connectivity/routing problems with their server. In some cases, routing appears to be working in one direction, yet broken in another direction. In other cases some services appear to work from some subnets, but fail from other subnets. In each of these cases, routing was never changed and is configured completely and properly.

Development fixed an issue with the rp_filter kernel setting. The default, global setting on SLES distributions is to have this set to mode 1, or strict setting. Prior to support pack 1 this setting was not enforced properly. Now that it is functioning properly Multi-homed servers or environments with asymmetric routing may be affected.

SLES 11 documentation (section 8.2): https://www.suse.com/releasenotes/x86_64/SUSE-SLES/11-SP1/#update-from-sles11ga

Resolution

Edit the /etc/sysctl.conf file and change the mode to disabled for the following line (add if necessary):

net.ipv4.conf.all.rp_filter = <mode>

0 - Disabled
1 - Enabled/Strict (default)

2 - Enabled/Loose

(see the Additional Notes section below for more details)
Run the following command:
sysctl -p

NOTE: While the default setting for rp_filter is 0, or disabled, some systems have been reported to have this set to either 1 or 2 under the specific interface. Double-check that rp_filter is also set to 0 under /proc/sys/net/ipv4/conf/*. If necessary, add cooresponding entries into the sysctl.conf to ensure the setting remains disabled (or 2, enabled/loose) upon reboot.

This statement doesn't mean that rp_filter is not enabled on SLES 10 and 11 distros. It is globally set with the net.ipv4.conf.all.rp_filter setting in the sysctl.conf as described above.

As per the documentation the max value from conf/{all,interface}/rp_filter is used when doing source validation on the {interface}. For example, if the net.ipv4.conf.all.rp_filter is set to 0, and net.ipv4.conf.etho.rp_filter is set to 1 or 2, then the highest setting (1 or 2) will take effect.

Additional Information

In SLES11 GA the "Reverse Path Filtering" setting used by the kernel (and found in the /proc/sys/net/ipv4/conf/all/rp_filter file) was, by default, set to 1 but the kernel was not acting upon the setting. In SLES11 SP1 development fixed the Kernel code to properly use this setting.　

The SLES 11 GA kernel was not performing the Reverse Path Filter function even though the setting was set.　

When development fixed this deficiency and released that fix in SLES 11 SP1 the kernel began validating all incoming packets based on the setting.

As an Example: If a packet was not coming in through the most direct path it would now be dropped.

Reverse Path Filtering is now working as designed.

rp_filter - INTEGER

0 - No source validation.

1 - Strict mode as defined in RFC3704 Strict Reverse Path

Each incoming packet is tested against the FIB and if the interface

is not the best reverse path the packet check will fail.

By default failed packets are discarded.

2 - Loose mode as defined in RFC3704 Loose Reverse Path

Each incoming packet's source address is also tested against the FIB

and if the source address is not reachable via any interface

the packet check will fail.

Current recommended practice in RFC3704 is to enable strict mode

to prevent IP spoofing from DDos attacks. If using asymmetric routing

or other complicated routing, then loose mode is recommended.

The max value from conf/{all,interface}/rp_filter is used

when doing source validation on the {interface}. (Which means it can be set

on each individual interface, or globally. The global setting will override the

individual interface settings).

This is globally set to mode 1 on on SLES 10 and 11 distributions

NOTE: The Suse Firewall may change this despite what is set inside of the sysctl.conf. This is due to a configuration theme enabled under the /etc/sysconfig/SuSEfirewall2 script. Search for FW_KERNEL_SECURITY="yes". The paragraph or two above this setting describe in detail what it is, and what it affects. Set this option to "no". The scope of this TID does not cover the ramifications of changing this setting. The individual settings affected by this change can be manually configured via the sysctl.conf.

TIP: A simple netstat -as | grep -i IPReversePathFilter will show whether or not the reverse path code is dropping packets.

This counter is 0 when the server first boots up.

If it has a value greater than 0 then the reverse path filter code has dropped packets since the last reboot.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.