Debugging or running another instance of rsyslog

SUSE Linux Enterprise Server 15
SUSE Linux Enterprise Server 12

Debugging or running another instance of rsyslog could be problematic because since SLES 12 the system logger is journald (rsyslog is a kind of attendant system logger providing log files expected from old times); thus there could occur an issue where both loggers try to usurp /dev/log device.

Debugging "system" rsyslog while still being under control of systemd

WARNING: It is impossible to use /run/systemd/journal/syslog socket managed by 'syslog.socket' unit without running 'rsyslog.service' unit (that is "system" rsyslog daemon). It is because 'syslog.socket' needs the referring service to be bound, otherwise it will not start. Thus, it is not possible to run rsyslog from shell in foreground with '-d' option and expect it would obtain data from journald (assuming default 'ForwardToSyslog=yes' is still effective, see journald.conf(5) for details). Moreover, if rsyslog is not running under systemd control, that is not running as systemd service unit, it may usurp /dev/log device which is symlink to /run/systemd/journal/dev-log by default and used by journald. Thus, running "system" rsyslog manually from shell is very not recommended.

Debugging rsyslog via environment variables

NOTE: Do not use '-d' inside RSYSLOG_PARAMS in /etc/sysconfig/syslog. By default debugging data is sent to standard output, but that is suppressed, by default, with 'StandardOutput=null' in rsyslog.service unit definition.

If rsyslog documentation How to create a debug log does not suite your needs, then to use a more general debugging, you should use environment variables.

First, edit /etc/sysconfig/syslog and add RSYSLOG_DEBUGLOG and RSYSLOG_DEBUG environment variables with their appropriate values. See Rsyslog Debug Support documentation for more details.

# grep -Pv '^\s*(#|$)' /etc/sysconfig/syslog 
RSYSLOGD_PARAMS=""
RSYSLOG_DEBUGLOG="/tmp/rsyslogd.txt"
RSYSLOG_DEBUG="Debug"

# systemctl restart rsyslog

Then, '/tmp/rsyslogd.txt' used in this example, would revel debugging info:

# head /tmp/rsyslogd.txt 

2539.782894619:main thread    : rsyslogd.c: rsyslogd 8.2106.0 startup, module path '', cwd:/
2539.782979970:main thread    : glbl.c: rsyslog/glbl: using '127.0.0.1' as localhost IP
2539.783023943:main thread    : obj.c: caller requested object 'net', not found (iRet -3003)
2539.783040624:main thread    : modules.c: Requested to load module 'lmnet'
2539.783052356:main thread    : modules.c: loading module '/usr/lib64/rsyslog/lmnet.so'
2539.783172582:main thread    : modules.c: module lmnet of type 2 being loaded (keepType=0).
2539.783186007:main thread    : net.c: entry point 'isCompatibleWithFeature' not present in module
2539.783193742:main thread    : net.c: entry point 'setModCnf' not present in module
2539.783201225:main thread    : net.c: entry point 'getModCnfName' not present in module

 

Running another (non-system) instance of rsyslog

NOTE: Enabling imuxsock.so module in rsyslog (it is enabled by default), would cause an additional rsyslog instance to usurp /dev/log device which is used by systemd-journald. If the another instance does not need to bind a local unix socket, then disable imuxsock.so module in rsyslog configuration file. If the another instance needs to bind a unix socket, then replace '$ModLoad imuxsock.so ' with modern syntax definition and with explicit unix socket path. An example: module(load="imuxsock" SysSock.Name="<path to the unix socket>").
 

Making systemd controlled rsyslog instance

Systemd service unit instance file is needed (but with some unit options removed as these are needed only for "system" rsyslog service). Moreoever, the new unit file requires /etc/sysconfig/rsyslog-<instance> - '%' refers to the instance name, see systemd.unit(5) for details - to start).

An example of an additional rsyslog service instance unit file:

# grep -Pv '^\s*#' /etc/systemd/system/rsyslog@.service
[Unit]
Description=Rsyslog %i Logging Service
Wants=network.target network-online.target
After=network.target network-online.target
Documentation=man:rsyslogd(8)
Documentation=http://www.rsyslog.com/doc/

[Service]
Type=notify
Environment=RSYSLOGD_PARAMS=
EnvironmentFile=/etc/sysconfig/rsyslog-%i
ExecStart=/usr/sbin/rsyslogd -n -iNONE $RSYSLOGD_PARAMS
ExecReload=/bin/kill -HUP $MAINPID
StandardOutput=null
Restart=on-abort

LimitNOFILE=16384

[Install]
WantedBy=multi-user.target

Then, systemd needs to be told to take this file into account.

# systemctl daemon-reload

Thus to make such rsyslog instance effective, /etc/sysconfig/rsyslog-<instance> must be adjusted. An example:

# cat /etc/sysconfig/rsyslog-test1
RSYSLOGD_PARAMS="-f /etc/rsyslog-test1.conf"

An example for rsyslog 'test1' instance configuration:

# cat /etc/rsyslog-test1.conf 
module(load="immark")
module(load="imudp")
input(type="imudp" port="5140")
module(load="imtcp")
input(type="imtcp" port="5140")
action(type="omfile" dirCreateMode="0700" FileCreateMode="0644"
       File="/var/log/remote-messages")

Discussing rsyslog config file is outside this TID, please refer to rsyslog documentation.

# systemctl start rsyslog@test1.service
# systemctl is-active rsyslog@test1.service
active
# systemctl is-enabled rsyslog@test1.service
disabled
# systemctl enable rsyslog@test1.service
Created symlink /etc/systemd/system/multi-user.target.wants/rsyslog@test1.service → /etc/systemd/system/rsyslog@.service

A quote from https://www.freedesktop.org/wiki/Software/systemd/syslog/ :

Note that it is now the journal that listens on /dev/log, no longer the BSD syslog daemon directly. If your logging daemon wants to get access to all logging data then it should listen on /run/systemd/journal/syslog instead via the syslog.socket unit file that is shipped along with systemd. On a systemd system it is no longer OK to listen on /dev/log directly, and your daemon may not bind to the /run/systemd/journal/syslog socket on its own. If you do that then you will lose logging from STDOUT/STDERR of services (as well as other stuff).

Rsyslog documentation:

• https://www.rsyslog.com/doc/v8-stable/troubleshooting/howtodebug.html
• https://www.rsyslog.com/doc/v8-stable/troubleshooting/debug.html

Systemd documentation:

• https://www.freedesktop.org/wiki/Software/systemd/syslog/