idmapd errors about "localdomain", or chown failing on nfs4 mount, with "invalid argument"

This document (7014266) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 15

SUSE Linux Enterprise Server 12

SUSE Linux Enterprise Server 11

Situation

An NFS client using an NFS4 (NFSv4) mount is having trouble with user identities, and logs errors such as:

rpc.idmapd[18466]: nss_getpwnam: name 'root@domain.com' does not map into domain 'localdomain'

Or when trying to do a "chown", it may show:

chown: changing ownership of `filename': Invalid argument

Resolution

The cause of this and the history of the concepts involved are complex. There are two general ways to correct this. The simplest way will be listed here, but it may not be possible on older kernels, and it may not be an ideal solution for all environments. Therefore, it is recommended to review the "Cause" section of this document before deciding on the best solution.

The simplest approach is to disable the NFS client's use of NFS Identity Mapping with the setting shown below. On SLES 11 SP2-SP4 the "disable" setting is available but is not on by default, so it may need to be turned on to disable the feature. On SLES 12 and 15, the setting is already on (feature disabled) by default. That means this kind of issue is less likely to appear on SLES 12 and 15. However, depending on the SLES configuration or the behavior of 3rd party NFS systems with which SLES is interacting, some concerns may still exist.

Set the following parameter for the kernel nfs module:

nfs.nfs4_disable_idmapping=1

That parameter can be set a variety of ways:

A. It can be done via Yast --> System --> Boot loader, by adding the kernel command line option:
nfs.nfs4_disable_idmapping=1

B. Alternatively, it can take effect slightly later during boot if the following has been done:

Edit or create /etc/modprobe.d/99-nfs.conf
and add (or modify) the line:
options nfs nfs4_disable_idmapping=1"

C. It can also be set on-the-fly (in case immediate reboot is not possible) with:

echo 1 > /sys/module/nfs/parameters/nfs4_disable_idmapping

Note, however, that setting this on the fly will only impact mounts performed after it is set. It will be necessary to umount and remount existing nfs4 mounts to put it into effect there. It is best to umount all mounts first, before remounting any, because multiple mounts pointing to one server can share an environment. Umounting is not always possible (if the file systems are busy) so scheduling a reboot may be needed anyway.

See the "Cause" section for other options, though likely less preferred.

Cause

When NFSv4 was originally conceived and implemented, it handled user identities differently than NFSv3. In v3, an nfs client giving a user's identify would simply pass a UID number in chown (and other requests) and the nfs server would accept it, even if the nfs server did not know of an account with that UID number. However, v4 was originally designed to pass identities in the form of <username>@<id-map-domainname>. To function correctly, that normally requires idmapd (id mapping daemon) to be active at client and server, and for each to consider themselves part of the same id mapping domain.

Chown failures or idmapd errors like the ones documented above are typically a result of either:

1. The username is known to the client but not known to the server, or

2. The idmapd domain name is set differently on the client than it is on the server.

Therefore, this issue can be fixed by insuring that the nfs server and client are configured with the same idmapd domain name (on Linux, this is done within /etc/idmapd.conf) and both have the same knowledge of the usernames / account details.

More detailed information about troubleshooting idmapd can be found in TID 7005060, which is typically available at either of these links:

https://www.suse.com/support/kb/doc/?id=7005060
https://www.suse.com/support/kb/doc/?id=000017244

However, it is often not convenient to insure that both sides (client and server) have the same user account knowledge, especially if the NFS Server is a filer. Therefore, troubleshooting and fixing the idmapd configuration is often not the preferred or quickest solution. For that, refer back to the "Resolution" section. However, in some cases, an old device with an old OS is present, which cannot disable the use of idmapping with NFS v4.

Additional Information

The NFS community has recognized that the idmapd feature of NFSv4 is often more trouble that it is worth, so NFSv4 has been modified allow the UID/GID method (already used in NFSv3) to work even under NFSv4.

The "Resolution" section above was written from the perspective of returning to traditional UID/GID identify methods (already used in NFS v3) even when using NFS v4. However, it may not be possible in all cases (especially with older kernels) so the following information may be helpful:

NFS Clients:
Machines running SLES 11 SP2 or higher, and acting as NFS clients, will have the ability to use the UID/GID identity behavior under NFSv4, but it is not necessarily the default behavior. Setting the kernel module parameter as described in the "Resolution" section of this document is needed. In mainstream Linux, the default behavior changed in kernel 3.3, i.e. to already have nfs4_disable_idmapping set to 1. So, for example, on SLES 12 machines, setting this manually is not needed (unless it has been manually set to 0, previously)

NFS Servers:
Some NFSv4 Servers may not be prepared to accept the UID/GID method. Both the NFS Client machine and the NFS Server machine need to be new enough to accept that method in v4. If using a Linux NFSv4 Server, it is necessary to use a distribution with kernel 3.4 or higher (for example, SLES 12). For 3rd party filers, check the NFS configuration for identity settings related to ID mapping, UIDs, etc.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.