kernel: nfsd: too many open TCP sockets, consider increasing the number of threads
This document (7010903) is provided subject to the disclaimer at the end of this document.
Environment
SUSE Linux Enterprise Server 15
SUSE Linux Enterprise Server 12
SUSE Linux Enterprise Server 11
SUSE Linux Enterprise Server 12
SUSE Linux Enterprise Server 11
SUSE Linux Enterprise Server 10
Situation
A busy NFS server may occasionally log the following message in /var/log/messages:
kernel: nfsd: too many open TCP sockets, consider increasing the number of nfsd threads
Resolution
The threads in question are set in /etc/sysconfig/nfs:
USE_KERNEL_NFSD_NUMBER="4"
The default is 4 but can be raised to allow nfsd to handle more requests at once. Based on experience with many cases, SUSE support gives the following guidelines:
Boosting the number of threads (even dramatically) is not likely to cause any negative side effects unless the machine in question is already struggling to keep up with workload even before any NFS work is introduced. Therefore, it is safe to be generous with this setting.
When putting any NFS Server into enterprise production, it would be good to boost this setting to at least 16. This will give it more room for load than the default of 4.
Then, if NFS client performance problems are seen or suspected, experiment with values in the range of 32 to 128. Something within this range will usually prove to be both safe and sufficient for busy enterprise servers. In some cases of extreme load, going even higher than 128 may be needed.
After making a change, the nfs server daemon can be restarted with:
ON SLES 12 or 15:
systemctl restart nfsserver
On SLES 10 or 11:
Boosting the number of threads (even dramatically) is not likely to cause any negative side effects unless the machine in question is already struggling to keep up with workload even before any NFS work is introduced. Therefore, it is safe to be generous with this setting.
When putting any NFS Server into enterprise production, it would be good to boost this setting to at least 16. This will give it more room for load than the default of 4.
Then, if NFS client performance problems are seen or suspected, experiment with values in the range of 32 to 128. Something within this range will usually prove to be both safe and sufficient for busy enterprise servers. In some cases of extreme load, going even higher than 128 may be needed.
After making a change, the nfs server daemon can be restarted with:
ON SLES 12 or 15:
systemctl restart nfsserver
On SLES 10 or 11:
rcnfsserver restart
Cause
If the NFS Server has a lot of requests coming in, the default 4 threads will not likely keep up with the workload. The message in the "Situation" section above is generated by some very basic rule-of-thumb checking, and is not a great indicator of when performance problems might begin to happen. It does not issue the warning based on actual work being done, but only based on the number of NFS connections present.
The message is given when connections to the NFS server's port 2049 are greater than:
[(number of threads + 3) * 20]
For example, at the default of 4, the message will be logged if connections to port 2049 exceed 140. However, there could be many connection present with each doing very little work, or there could fewer connections each doing a tremendous amount of work. Performance problems may exist even without the warning being logged.
The message is given when connections to the NFS server's port 2049 are greater than:
[(number of threads + 3) * 20]
For example, at the default of 4, the message will be logged if connections to port 2049 exceed 140. However, there could be many connection present with each doing very little work, or there could fewer connections each doing a tremendous amount of work. Performance problems may exist even without the warning being logged.
To decide how many threads are need to merely avoid the warning, it can sometimes be helpful to manually check the number of nfsd connections, especially over time or at peak work times. To do so, give the following command (at a terminal on the NFS Server).
netstat -nt | grep -c :2049 | wc -l
Then plug the trended count or peak count into the formula above and calculate the number of threads which would be needed to satisfy the algorithm.
However, even before reaching the number of connections which the formula indicates, performance problems related to the number of threads will often exist. Therefore, the recommendations given in the 'Resolution" section above are typically higher than would be indicated by the formula.
However, even before reaching the number of connections which the formula indicates, performance problems related to the number of threads will often exist. Therefore, the recommendations given in the 'Resolution" section above are typically higher than would be indicated by the formula.
Disclaimer
This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.
- Document ID:7010903
- Creation Date: 10-Oct-2012
- Modified Date:01-Oct-2021
-
- SUSE Linux Enterprise Server
For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback[at]suse.com
