NFS mount randomly fails when going through Cisco device with Firewall Services Module
This document (7016980) is provided subject to the disclaimer at the end of this document.
Environment
SUSE Linux Enterprise Server 11 Service Pack 3 (SLES 11 SP3)
Situation
An NFS server provides an export which is mounted on multiple NFS
clients.
Randomly one NFS client will loose it's connection to that export while other NFS clients are still able to access it.
Randomly one NFS client will loose it's connection to that export while other NFS clients are still able to access it.
Resolution
Resolution #1
This will most directly addresses the issue with the TCP Sequence Number Randomization feature, and is Cisco's recommendation:
Since TCP Sequence Number Randomization is a legacy feature that was supposed to protect hosts that use predictable algorithms for initial TCP sequence number generation, it is does not provide much additional security on the modern TCP stacks. Hence, the feature can be selectively disabled to take full advantage of TCP SACK and achieve the maximum throughput on a single TCP flow. The best way to disable the randomization is to use Modular Policy Framework (MPF); you can also narrow the class down just to those trusted hosts that do the high-speed transfers:
Resolution #2
Another recommendation if the Cisco device cannot be configured properly, is to disable TCP SACK on the end point servers.
Here are some commands to help you with this change.
Display the endpoint servers current TCP SACK setting
sysctl -a | grep -i tcp_sack
# my test server returns - net.ipv4.tcp_sack = 1
Change the value for this session:
sysctl -w net.ipv4.tcp_sack=0
# my test server returns - net.ipv4.tcp_sack = 0
This will not survive a reboot.
Reload sysctl
sysctl -p
This command reloads the settings in the /etc/sysctl.conf file which means that you could simply add this line to the /etc/sysctl.conf file so that on reboot, or a sysctl -p it would be loaded:
net.ipv4.tcp_sack = 0
Turning off the TCP SACK option will affect all future TCP connections with any other device.
In other words, TCP SACK will no longer be seen as an available TCP option for all TCP connections.
That will effectively work around the Cisco devices TCP Sequence Number Randomization issue with the TCP SACK option.
This will most directly addresses the issue with the TCP Sequence Number Randomization feature, and is Cisco's recommendation:
Since TCP Sequence Number Randomization is a legacy feature that was supposed to protect hosts that use predictable algorithms for initial TCP sequence number generation, it is does not provide much additional security on the modern TCP stacks. Hence, the feature can be selectively disabled to take full advantage of TCP SACK and achieve the maximum throughput on a single TCP flow. The best way to disable the randomization is to use Modular Policy Framework (MPF); you can also narrow the class down just to those trusted hosts that do the high-speed transfers:
class-map TCP
match port tcp range 1 65535
policy-map global_policy
class TCP
set connection random-sequence-number disable
service-policy global_policy global
Resolution #2
Another recommendation if the Cisco device cannot be configured properly, is to disable TCP SACK on the end point servers.
Here are some commands to help you with this change.
Display the endpoint servers current TCP SACK setting
sysctl -a | grep -i tcp_sack
# my test server returns - net.ipv4.tcp_sack = 1
Change the value for this session:
sysctl -w net.ipv4.tcp_sack=0
# my test server returns - net.ipv4.tcp_sack = 0
This will not survive a reboot.
Reload sysctl
sysctl -p
This command reloads the settings in the /etc/sysctl.conf file which means that you could simply add this line to the /etc/sysctl.conf file so that on reboot, or a sysctl -p it would be loaded:
net.ipv4.tcp_sack = 0
Turning off the TCP SACK option will affect all future TCP connections with any other device.
In other words, TCP SACK will no longer be seen as an available TCP option for all TCP connections.
That will effectively work around the Cisco devices TCP Sequence Number Randomization issue with the TCP SACK option.
Cause
NFS/TCP: TCP SACK handling is broken with seq randomization on Cisco Fire Wall Services Module
See this Cisco document for more information and recommendations:
https://supportforums.cisco.com/document/48551/single-tcp-flow-performance-firewall-services-module-fwsm
Specifically the section entitled: TCP Sequence Number Randomization and SACK
Disclaimer
This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.
- Document ID:7016980
- Creation Date: 12-Nov-2015
- Modified Date:03-Mar-2020
-
- SUSE Linux Enterprise Server
For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback[at]suse.com
