How to increase the inotify.max_user_watches and inotify.max_user_instances sysctls on a Linux host

The sysctls fs.inotify.max_user_instances and fs.inotify.max_user_watches define user limits on the number of inotify resources and inotify file watches. If these limits are reached, you may experience processes failing with error messages related to the limits, for example:

ENOSPC: System limit for number of file watchers reached...

The configured user limit (128) on the number of inotify instances has been reached 

The default defined inotify instances (128) has been reached

In the context of a Kubernetes cluster, this behaviour would exhibit as failing Pods, with inotify related errors in the Pod logs similar to the above. This article details how to check the current limits configured and how to increase these.

Check current limits

You can check the current inotify user instance limit, with the following:

cat /proc/sys/fs/inotify/max_user_instances

Similarly, the current inotify user watch limit can be checked as follows:

cat /proc/sys/fs/inotify/max_user_watches

Update the limits

You can update the limits temporarily, with the following commands (setting the values to 8192 and 524288 respectively in this example):

sudo sysctl fs.inotify.max_user_instances=8192
sudo sysctl fs.inotify.max_user_watches=524288
sudo sysctl -p

In order to make the changes permanent, i.e. to persist a reboot, you can set fs.inotify.max_user_instances=8192 and fs.inotify.max_user_watches=524288 in the file /etc/sysctl.conf.

After updating the limits, you can validate these on the host again, as above, with cat /proc/sys/fs/inotify/max_user_instances and cat /proc/sys/fs/inotify/max_user_watches.

To check the value as reflected in a running container, exec into the container and cat the files:

docker exec -it <CONTAINER ID> cat /proc/sys/fs/inotify/max_user_instances

and

docker exec -it <CONTAINER ID> cat /proc/sys/fs/inotify/max_user_watches

If the updated limits are not reflected on a host after running sysctl -p, reboot the host after setting the limits in /etc/sysctl.conf.