NFS4 mount shows all ownership as "nobody" or 4294967294

SUSE Linux Enterprise Server 15

SUSE Linux Enterprise Server 12

SUSE Linux Enterprise Server 11

SUSE Linux Enterprise Server 10

When the NFS client shows ownership of "nobody" but the NFS Server shows a different (usually more desired) ownership, this means that NFS 4 "id mapping" (governed by idmapd, the identity mapping daemon) is being attempted, but is not set up correctly.  ID mapping is an alternative to the method of simply sending UIDs and GIDs between client and server.

The identity mapping daemon is not used in NFS v3.  See the "Cause" section for more details on the purpose of idmapd, or for discussion of NFS v3 versus v4 in this regard.

NOTE, however, that correcting the idmapd configuration may not be the most desired solution.  The need for ID mapping was mandatory when NFS4 first came about, but now it is considered optional.  Adding ID mapping to an NFS configuration is generally considered complicated, unnecessary, and without real benefits.  Therefore, if possible, it is usually desired to avoid the use of ID mapping.

1.  Avoiding the idmapping method

1.1  Avoiding idmapping on Linux NFS Clients

A Linux NFS client will attempt (or not attempt) to use idmapd based on the nfs module option:

nfs.nfs4_disable_idmapping=1

This can be set to 0 or 1, off or on.  Note that this is a "disable" option, so having it ON (1) means that the use of idmapping will be disabled (or off).  However, even with idmapping "disabled", if the NFS client is interacting with a NFS Server with a very old implementation, the NFS Server might insist that idmapping be used, and then the client will fall back to using idmapping anyway.

SLES 12 and 15 (or likely anything using kernel 3.3 or higher) default to having this option set to "1", so those systems acting as NFS clients will not attempt idmapping by default.
SLES 11 (using kernel 3.0.x) has this option set to 0 by default, so it will need to be set to 1 if desired.
SLES 10 (using kernel 2.x) probably did not have this option at all.

The current value can be checked with:
cat /sys/module/nfs/parameters/nfs4_disable_idmapping

The option can be set a variety of ways:

A.  It can be done via Yast --> System --> Boot loader, by adding the kernel command line option:
nfs.nfs4_disable_idmapping=1
 
B.  Alternatively, it can take effect slightly later during boot if the following has been done:
Edit or create /etc/modprobe.d/99-nfs.conf
and add (or modify) the line:
options nfs nfs4_disable_idmapping=1"
 
C.  It can also be set on-the-fly (in case immediate reboot is not possible) with:
 
echo 1 > /sys/module/nfs/parameters/nfs4_disable_idmapping

Note, however, that setting it on the fly will only impact mounts performed after it is set.  It will be necessary to umount and remount existing nfs4 mounts to put it into effect there.  It is best to umount all mounts first, before remounting any, because multiple mounts pointing to one server can share an environment.

1.2  Avoiding ID mapping on Linux NFS Servers

A Linux NFS Server using kernel 3.4 or higher (including SLES 12 and 15) will allow clients to submit requests with either method:  ID mapping, or passing plain UID/GIDs.  So having NFS Servers "avoid" idmapping is simply a matter of being on a relatively modern kernel and then insure the clients avoid asking for it as well.

NFS Servers on older kernels (such as used on SLES 10 or 11) will insist on using idmapping.  An NFS 4 client which attempts to use the UID/GID method will be told to use idmapping instead.

1.3  Avoiding ID Mapping on non-Linux NFS Servers or Clients.

Many non-Linux NFS Servers or client may not automatically avoid using idmapping.  They may have configuration options which control whether NFS 4 will insist on using idmapping or will allow the old UID/GID method from v3 to be used on v4 as well.  It can simplify NFS v4 usage greatly to configure such 3rd party devices to allow the v3 method (UID/GID) to be used with v4 also.

2.  If or when ID Mapping is MANDATED

If ID mapping MUST be used, it is usually because either the NFS client or NFS server code (or both) is quite old.  In some cases, it is mandated by administrative decision.  In those cases, if the "nobody" ownership symptom (the original title and subject of this document) is present, it is typically caused when the NFS v4 identity mapping daemon (idmapd) is not running, or it is misconfigured, or it is holding old information in it's cache.

 

2.1   If idmapd not running.

 

2.1.1  SLES NFS Servers

SLES NFS Servers (those which export or "share" their file systems via NFS) need to be running idmapd in case a client requests that method.  On Linux, check for an idmapd process with:

 

 

A process for "/usr/sbin/rpc.idmapd" should be found.

 

On SLES 10, you can restart idmapd with the command "rcidmapd restart".

 

On SLES 11, 12, and 15, idmapd is not separately started, it is part of the initialization of the nfsserver service.  It should start any time "nfsserver" service is started.  So "rcnfsserver restart" can be used on SLES 11, or "systemctl restart nfsserver" on SLES 12 or 15.

2.1.2 SLES NFS Client machines

2.1.2.1  "Recent" SLES NFS client systems (SLES 12 SP4 and higher) typically do not require idmapd to be loaded at the client.  They will use an upcall program called "nfsidmap" to accomplish the client functions of idmapping.

2.1.2.2   Earlier NFS 4 clients (older code found in SLES 10, 11, and early 12 support packs) required idmapd to be running at the client machine as well.  On those and other older Linux distributions, check for an idmapd process with:

 

However, in some cases idmapd might not start.  For example, if the /etc/fstab does not contain any NFS v4 mount, it is possible that idmapd will not be started even when the "nfs" service starts.

 

2.2.  If idmapd is misconfigured

 

For correct idmapping between NFS clients and NFS servers, both must have good /etc/idmapd.conf settings.  Even when the same user accounts are known to both the servers and clients, idmapd configuration problems can prevent proper ownership from being displayed.

 

Check the /etc/idmapd.conf file.  The [General] section should have a Domain setting.  The domain is an arbitrary string but it must be set identically on NFS clients and their NFS servers.  This setting often reports "localdomain" by default, and that is usually adequate.  In many cases, administrators have set it to match the company's DNS domain name, but that is not required, and would technically be a coincidence, rather than accomplish anything meaningful.  The only requirement is that clients and servers use the idmapd domain as each other.

It can also be helpful for /etc/idmapd.conf to contain a [Translation] section which specifies the method of translating between names and IDs.  Typically, it is best to point to nsswitch methodology.

  

So, for example, a typical idmapd.conf file might look like the following.  If this file is changed, idmapd must be restarted.
 

[General]
Verbosity=7
Pipefs-Directory=/var/lib/nfs/rpc_pipefs
Domain=localdomain

[Mapping]
Nobody-User=nobody
Nobody-Group=nobody

[Translation]
Method=nsswitch

 

2.3.  If idmapd is caching old information.

 

Old information can be cached by idmapd.  This may temporarily prevent idmapd from learning new information.  Even the fact that a user has no mapping can be cached (still believed) beyond the point where newer information is created.  This is known as a "negative" cache.  The cache defaults to keeping entries for 10 minutes.  This is usually fine, because user identities do not change or get created very often.  However, idmapd may learn early in system boot that certain users were not found.  This might be due only to a delay in access to information.  For example, idmapd might start before sssd or other methods of obtaining user identities.  In such a case, negative cache might exist for the first 10 minutes after server boot.  To make the cache clear more quickly:

 

Stopping and starting idmapd will correct this, but that is not always desired.

 

On SLES 12 and higher, the command "nfsidmap -c" is available and should clear the cache and allow new information to be learned.

 

Another option is to lower the cache timer.  This is set inside /etc/sysctl.conf with:

 

fs.nfs.idmap_cache_timer = 60

(this value represents seconds, the default is 600)

 

When NFS 4.x was original conceived and implemented, a feature of ID mapping was created and mandated.  This concept had weaknesses both foreseen and unforeseen, so it has since become undesired and optional.  More detail of ID Mapping follows, for those less-common implementations which may still use it.  Note that this discussion may not fully apply to NFS over Kerberos.

If the ID mapping method is in use, the correct display of usernames depends upon the NFS v4 server having the same knowledge of user and group accounts (including their UIDs and GIDs) as the NFS v4 client is using, and both must be in the same idmapd domain.

 

When ID mapping is used, NFS v4 will pass identities between servers and clients in the form 'username@domainname'.

 

This is a major change from the traditional method of passing UID and GID numbers.  With the UID/GID method, an NFS server can store and report these UID values in the file system, even if it has no knowledge of the user accounts they belong to.  So essentially, a UID that was not known to one side or the other could still be handled in a valid way.  In contrast, ID mapping does not allow unknown identities to be preserved.  It will convert them to "nobody".

 

 

Put simply, in NFS v4 with idmapd, the same username@domainname must be recognized by both sides and represent the same UID, or it may be mapped to "nobody".

One of the original intentions of ID mapping (when it was conceived theoretically) was that the NFS client and NFS server would not need to get information from the same identity stores, and (for example) "userBOB@domainORG" would not need to be known as the same UID at both ends.  However, this theory did not keep in mind that the RPC layer (which all NFS depends upon) still focuses on UIDs and GIDs, so this goal is not currently obtainable.