journalctl --unit= -b output random lines are missing

This document (000019696) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 12
SUSE Linux Enterprise Server 15

Situation

journalctl might not be able to print all logging messages when using options to filter messages based on metadata.

For example:

When running 'journalctl -b | grep tftp', all output appears.
When running 'journalctl -b --unit=tftp', random lines are missing.

Resolution

Some options of journalctl might not be working correctly, especially those which are used to match entries based on process metadata, such as"--unit=".

For this reason it's advised to use "journalctl | grep <pattern>" instead.
This is especially the case if the service is not a long-running one.

Cause

journald might not be able to retrieve metadata of the logging process if the process exits right after sending its messages as the info in /proc might be gone at the time journald receives the messages.

The consequence is that some of the process metadata normally added to each log entries might be missing in this case. However, the message itself is still recorded by journald.

Status

Reported to Engineering

Additional Information

This is known limitation: https://github.com/systemd/systemd/issues/2913
The missing messages are issued by processes that exited just (?) after emitting the missing logs.

This limitation has been mitigated in recent version of systemd but not completely solved. The definitive fix would need the kernel to send metadata along with the messages send to the socket but this works is still in progress.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.