How to monitor /dev/null for attribute changes

This document (7021124) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 11
SUSE Linux Enterprise Server 12
audit

Situation

An unknown process turns /dev/null from a character device into a regular file. /usr/bin/lsof -b +M -n -l shows processes complaining /dev/null being deleted:

[...]
systemd       1              0    2   unknown                          /dev/null (deleted) lstat: Resource temporarily unavailable) (stat: Resource temporarily unavailable)
dmeventd    649              0    0   unknown                          /dev/null (deleted) lstat: Resource temporarily unavailable) (stat: Resource temporarily unavailable)
haveged     656              0    0   unknown                          /dev/null (deleted) lstat: Resource temporarily unavailable) (stat: Resource temporarily unavailable)
lvmetad    1773              0    0   unknown                          /dev/null (deleted) lstat: Resource temporarily unavailable) (stat: Resource temporarily unavailable)
nscd       1899 1901      496    2   unknown                          /dev/null (deleted) lstat: Resource temporarily unavailable) (stat: Resource temporarily unavailable)
[...]

The goal is to monitor /dev/null to identify the accessing program.

Resolution

To recreate /dev/null please run the following commands:

rm /dev/null
mknod /dev/null -m 666 c 1 3

The Linux kernel auditing daemon provides a way to monitor attribute changes for given paths. For the monitoring to take place, please install the package audit and enable the daemon:

SLES 11 SP4:
insserv auditd
rcauditd start

SLE 12:
systemctl enable auditd
systemctl start auditd #(rcauditd <option> would work as well)

A non reboot persistent monitor may be set up easily using:

auditctl -w /dev/null -p a -k <monitorname>

For <monitorname> just pass a name that will help to identify the entries in the log file, e.g:

auditctl -w /dev/null -p a -k nullmonitor

If a reboot persistent solution is required, please edit /etc/audit/audit.rules and add the rule here:

# Feel free to add below this line. See auditctl man page

-w /dev/null -p a -k nullmonitor

When done, reload the auditd daemon:

rcauditd reload

Monitor /var/log/audit/audit.log for occurrences of the key "nullmonitor" and identify the process changing /dev/null.

Additional Information

A possible testcase

Create a character device to monitor. For this use case the device /dev/nulltest will be created:

mknod /dev/nulltest -m 666 -c 1 100
sles12sp2:~ # ls -l /dev/nulltest
crw-rw-rw- 1 root root 1, 100 Jul 27 11:57 /dev/nulltest

Note the c at the start which identifies /dev/null as a character device.
In order to avoid conflicts with already existing charater devices, please lookup existing character device numbers in /sys/dev/char/1\:*
This will show a list of character devices currently managed by udev.
Make sure the audit daemon was started and setup the monitor rule:

rcauditd status
auditctl -w /dev/nulltest -p a -k nullmonitor
Now let's do something to /dev/nulltest:

cd /root
touch nulltest
mv /root/nulltest /dev/
sles12sp2:~ # ls -l /dev/nulltest
-rw-r--r-- 1 root root 0 Jul 27 11:58 /dev/nulltest

So we turned /dev/nulltest from a character device into a regular file.
/var/log/audit/audit.log shows:

rizzo:~ # grep nullmonitor /var/log/audit/audit.log
type=CONFIG_CHANGE msg=audit(1501148396.172:3): auid=0 ses=1 op="add_rule" key="nullmonitor" list=4 res=1
type=CONFIG_CHANGE msg=audit(1501148479.964:4): auid=0 ses=1 op="updated_rules" path="/dev/nulltest" key="nullmonitor" list=4 res=1
type=CONFIG_CHANGE msg=audit(1501148479.964:5): auid=0 ses=1 op="updated_rules" path="/dev/nulltest" key="nullmonitor" list=4 res=1
type=SYSCALL msg=audit(1501148479.964:6): arch=c000003e syscall=190 success=yes exit=0 a0=4 a1=7f00620200df a2=1a09080 a3=1c items=1 ppid=2304 pid=2843 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=1 comm="mv" exe="/usr/bin/mv" key="nullmonitor"

Here we see /usr/bin/mv accessed /dev/nulltest. The syscall executed may get identified using ausyscall <arch> <number>, e.g. ausyscall x86_64 190:

sles12sp2:~ # ausyscall x86_64 190
fsetxattr

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.