Configure SAP services to log outsite of journald

Why should we do this – The story behind

Journald is an integral part of systemd. It is the centralized location for all messages logged by different components in a modern linux system. On a SUSE system all the messages are forwarded in addition to rsysylog. This means all logs are stored in systemd-journal and on a traditional logfile as well. But what is if we don’t want to have special logs in journald, maybe because of some security reason. In our, maybe not too illusory, scenario we have an admin who is allowed to read the normal journald logs but not any files under /var/log/. SAP HANA database is installed and configured to write audit policies to system log (https://me.sap.com/notes/2624117). Therefore confidential information are in the journal of the OS. The question is now, how can we avoid having those confidential information in the journal?

LogNamespace is our friend

Starting with systemd version 245 the feature LogNamespace was indroduced. It gives us the possibility to define a own, independent log stream managed by its own instance of systemd-journald. So all we need to do is, to define a LogNamespace, in the service file we want to have the new log stream, and tell the new systemd-journald instance not to log the messages itself  but forward it to rsyslog (where, in our scenario the admins don’t have any rights).

There is however one small problem. The LogNamespace (including the socket) will be created not before the service file is started. And because rsyslog is starting much earlier on boot we will get an error on creating or binding socket. This is  because the socket is not available at that point of time. The solution here is a small help-service file which create the LogNamespace before the start of the rsyslog daemon.

The implementation

The implementation is quite easy. All we have to do is to create three files. Our helper-service file, an override file for the HANA systemd service and an include file for rsyslog.

Create the help-service file

The only reason for this file is to create the LogNamespace before rsyslog.service is starting.
So we can get the proper socket file, we would use later on in the rsyslog config.

# vi /etc/systemd/system/activate-LogNamespace_HANALog.service
[Unit]
Description=Create LogNamespace HANALog before rsyslog start
Wants=rsyslog.service

[Service]
LogNamespace=HANALog
ExecStart=/usr/bin/echo "Creating LogNamespace HANALog..."

[Install]
WantedBy=multi-user.target

Create a drop-in file

Let’s assume our SAP HANA service file is  SAPTS0_01.service. We simple gave it the LogNamespace HANALog.
We can use the edit option of systemctl to create an override file for our changes.

 # systemctl edit SAPTS0_01.service
 [Service]
 LogNamespace=HANALog

The created override file is then:

 /etc/systemd/system/SAPTS0_01.service.d/override.conf

Configure the LogNamespace

A  LogNamespace instance can be configured through /etc/systemd/journald@NAMESPACE.conf. So we can configure our own instance individually.
In our example, we want to forward all the logs to rsyslog only. This means we have to set Storage to none in addition.

 # vi /etc/systemd/journald@HANALog.conf
 [Journal]
 ForwardToSyslog=yes
 Storage=none

Adapt rsyslog to listen to that namespace

On default rsyslog is listening on /run/systemd/journal/syslog. To get the HANALog stream into rsyslog we have to create an extra include file where we define the source /run/systemd/journal.HANALog/syslog in addition:

 # vi /etc/rsyslog.d/HANALog.conf
 $ModLoad imuxsock.so
 input(type="imuxsock" Socket="/run/systemd/journal.HANALog/syslog")

As an optional configuration, we can re-route all the audit logs (starting with HDB..)  to an extra file.

 if ($programname startswith 'HDB')
 then {
        -/var/log/auditredirect.log
        stop
 }

Reload and restart all the services

As a last step the systemd daemon should be reloaded, the helper-service has to enable and start once and rsyslog should be restarted. With that we get all the changes adopted.

# systemctl daemon-reload
# systemctl enable --now activate-LogNamespace_HANALog.service 
# systemctl restart rsyslog.service

Please keep in mind that especially on a high log message rate, the restart of rsyslog can case lost of some messages.

Summary

Even if the solution above is using the journal in the first place, it is still possible to have special logs in a separate file only without having anything in the journal.
The example above should only give some inspiration what can be done with journald and rsyslog in general.