How to troubleshoot rancher-logging

This document (000020988) is provided subject to the disclaimer at the end of this document.

Situation

After installing the rancher-logging app and creating your resources, flows(clusterflows)/outputs(clusteroutputs), how do you troubleshoot if you are not seeing your logs at the final destination?

Resolution

The rancher-logging app troubleshooting can be divided into three phases as below:

User resources review: The first part is to ensure there is no error for the resources you created and that they are active.

1. ClusterOutputs and Outputs review. They should be active with no errors. Please correct the errors if any

kubectl get clusteroutput -A
 NAMESPACE   NAME            ACTIVE   PROBLEMS
 test-ns     test-c-output   true
kubectl get output -A
 NAMESPACE   NAME          ACTIVE   PROBLEMS
 test-ns     test-output   true

2. Clusterflows and flows review. They should be active with no errors. Please correct the errors if any

kubectl get clusterflow -o wide -A
 NAMESPACE   NAME          ACTIVE   PROBLEMS
 test-ns     test-c-flow   true
kubectl get flow -o wide -A 
 NAMESPACE   NAME        ACTIVE   PROBLEMS
 test-ns     test-flow   true

FluentD and Fluentbit pods review: The fluentbit is a daemonset and should be running on each node, while you should have at least one FluentD pod running as the fluentbit pods will collect logs from each node and forward them to the FluentD pod to be sent to their final destination

Your output could look different depending on what type Kubernetes cluster you have. However, you should have a fluentbit pod on each node and at least one FluentD pod

kubectl get pods -n cattle-logging-system
NAME                                                READY   STATUS      RESTARTS   AGE
rancher-logging-655578478b-7k46r                    1/1     Running     0          89s
rancher-logging-k3s-journald-aggregator-957gz       1/1     Running     0          89s
rancher-logging-root-fluentbit-lczl5                1/1     Running     0          70s
rancher-logging-root-fluentd-0                      2/2     Running     0          70s
rancher-logging-root-fluentd-configcheck-ac2d4553   0/1     Completed   0          84s

Logs review: At this step, you review to ensure no errors in the logs for the FluentD or Fluentbit pods. For Fluentbit, you will probably need to review each if you suspect the logs are not being collected from fluentbit.

1. kubectl exec rancher-logging-root-fluentd-0 -n cattle-logging-system -- cat /fluentd/log/out  # logging-operator version +up3.17.10 and earlier 
   kubectl logs rancher-logging-root-fluentd-0 -n cattle-logging-system #  logging-operator version +up4.4.0 and later 

This will dump the logs out of the fluentd container

2. kubectl -n cattle-logging-system logs rancher-logging-root-bit-lczl5

You should run it against each fluentbit pod if you suspect that the issue is on fluentbit

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.