Logs not forwarded by Rancher Logging in Rancher v2.x when Docker daemon logging driver is not set to json-file

The Rancher v2.x Logging feature enables you to configure log forwarding for Pods, as well as system component containers, in a cluster to a logging endpoint such as Elasticsearch or Splunk.

This feature works by deploying a workload to each node in the cluster that mounts the container log directory from the host to parse the Docker container json log files. This is dependent upon use of the json-file Docker logging driver. In the event that the Docker daemon is configured with an alternative logging driver, the logging feature will be unable to parse the logs and will not forward these.

However, under certain configurations (e.g., in CentOS and RHEL packaged Docker 1.13.1, the default log driver configured is journald), it could prevent log forwarding functioning. Meanwhile, whilst json-file is the default log driver in the upstream Docker packages, if an alternative has been configured on nodes this will also prevent the correct functioning of the log forwarding.

You can verify the currently configured Docker logging driver on a node by running docker info | grep Logging, which will show output of the following format: Logging Driver: journald.

In the event that json-file is not the configured logging driver, the output of ls -la /var/log/containers/ on the node should also be empty. With json-file configured this would display symoblic links to paths under /var/log/pods, containing symbolic links which in turn point to the Docker container json log files.

CentOS or RHEL packaged Docker

1. Update /etc/sysconfig/docker, to set --log-driver=json-file instead of journald.
2. Restart the Docker daemon: systemctl restart docker
3. You should now see symlinked logs created under /var/log/containers

Upstream Docker

1. Configure the json-file Docker logging driver in /etc/docker/daemon.json per the Docker documentation
2. Restart the Docker daemon: systemctl restart docker
3. You should now see symlinked logs created under /var/log/containers