NFS mount attempts fail with "permission denied" or succeed initially but fail after 30 minutes

SUSE Linux Enterprise Server 15 SP3
SUSE Linux Enterprise Server 15 SP2
SUSE Linux Enterprise Server 15 SP1
SUSE Linux Enterprise Server 15 SP0

NFS v4 client mount attempts against a Linux NFS Server fail immediately with "permission denied".

NFS v3 client mount attempts against a Linux may fail immediately, or may succeed but after 30 minutes stop working, with "permission denied".

There are, of course, many reasons an NFS Server could return "permission denied," but for this particular scenario, several unique factors and clues are present.

1.  The possible difference in v3 behavior as described above.

2.  When checking certain output at the Linux NFS Server machine, the following error may be seen:
conf_parse: last line non-terminated, ignored.

Places on the Linux NFS Server machine where this message have been noted include:
a.  Output of "exportfs -v"
b.  /var/log/messages
c.  Output of "systemctl status nfs-server" (but not from "systemctl status nfsserver")
d.  Output of "journalctl"

3.  After "permission denied" is given at the client machine, issuing the following command at the NFS Server machine:
cat /proc/net/rpc/auth.unix.ip/content

will show a line similar to:
nfsd 192.168.1.245 -no-domain-

It is the "-no-domain-" part of that line which is of unique interest here.  This indicates that something has gone wrong with the way NFS clients are being trusted (allowed) to mount what the NFS Server is exporting.

4.  When "permission denied" is received from the NFS Server, packet traces show that the RPC header (not NFS header) of the reply packet includes the following (as viewed from wireshark):
AUTH_ERROR (1)
bad crediential (seal broken) (1)

The problem can be fixed at the NFS Server machine, either manually in configuration files or by updated code which avoids the problem.

1.  To fix manually (recommended, even if code will also be updated):

The last line of certain NFS-related configuration files may be unterminated.  In other words, the file does not end with a "new line" character.  This is quite unusual, as most Linux based editors will automatically include a new line at the end of an ASCII file when it is typed up and saved, even if the user does not explicitly press <enter> after the final line.  As such, it is probably best to correct this condition if it is found.  Files where this condition may trigger this problem include:

/etc/nfs.conf.local  #which may not exist, but (if it does) is the most likely source of the problem
/etc/nfs.conf
/etc/sysconfig/nfs

Edit those files.  If, while editing the file, it is already possible to move down to a final empty line, then nothing needs to be changed.   However, if the file ends at the right hand side of a non-empty line (even in a comment line) this can trigger the problem.  After the last character of the last line, insert a new line by pressing the <enter> key.  If using "vi", be sure to be in "insert mode" first.

You could also check those files for new-line characters with the "od" command.  For an example, see the "Additional Information" section below.

2.  To fix this with updated code:

The "nfs-kernel-server" package needs to be updated.  Incidentally, this package is built from a source code package named "nfs-utils," so it is best to update the "nfs-client" package at the same time, as both are built from the "nfs-utils" source package.

On SLES 15 SP1 / SP2 / SP3, the fix was first supplied in nfs-kernel-server 2.1.1-10.21.1.  For SP3, this is available in general maintenance updates.  However, for SP1 and SP2 it is available only with LTSS or ESPOS entitlement.

On SLES 15 without any support pack, the fix was first supplied in nfs-kernel-server 2.1.1-6.17.1 (available only with LTSS or ESPOS entitlement).

Interestingly, the problem is only indirectly caused by the lack of line termination.  When the lack of line termination is detected, the mountd (the mount daemon) sends a message to the logging daemon warning "conf_parse: last line non-terminated, ignored."  This message causes a socket to be opened to the logging daemon, and it is left open.  When closeall() is subsequently called, the socket in closed but the logging library thinks it is still open.  Later, this causes confusion and mountd ends up blocking while trying to read from something that isn't what it thinks it is.

Upstream fix is
Commit 7fc4d064f951 ("mountd: Initialize logging early.")

In the upstream public project for nfs-utils, this fix first came in nfs-utils version 2.4.2.  However, within SLES, it has been backported into the earlier versions listed above in the "Resolution" section.
 

Pursuant to the manual fix in the "Resolution" section above, it is possible to check for end-of-line characters more certainly and without opening the files in an editor.  For example, the "od" command can be used.  Consider a /etc/nfs.conf.local file which to the naked eye appears as the two following lines:

# End of /etc/nfs.conf.local
############################

Using this command to view it:

od -cb nfs.conf.local
0000000   #       E   n   d       o   f       /   e   t   c   /   n   f
        043 040 105 156 144 040 157 146 040 057 145 164 143 057 156 146
0000020   s   .   c   o   n   f   .   l   o   c   a   l  \n   #   #   #
        163 056 143 157 156 146 056 154 157 143 141 154 012 043 043 043
0000040   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #
        043 043 043 043 043 043 043 043 043 043 043 043 043 043 043 043
0000060   #   #   #   #   #   #   #   #   #
        043 043 043 043 043 043 043 043 043
0000071

In the od output above, the end of the file's first line is designated with the "\n" symbol, aka "new line", aka value 012.  That value may differ from 012 if other od options are used.  The final line has no such character at the end, meaning it is missing the new line character and could trigger the problem.