How to increase the log level of Kubernetes components in an RKE CLI or Rancher provisioned Kubernetes cluster

Task

When troubleshooting an issue with an RKE CLI or Rancher provisioned Kubernetes cluster, it may be helpful to increase the verbosity of logging on one or more of the Kubernetes components, above the default level. This article details the process of increasing logging on both those components that use the Kubernetes hyperkube image (kubelet, kube-apiserver, kube-controller-manager, kube-scheduler, kube-proxy) as well as the etcd component.

Pre-requisites

• A Kubernetes cluster provisioned by the RKE CLI or Rancher v2.x

Resolution

Kubernetes API Server, Controller Manager, Scheduler, Kube Proxy and Kubelet

The Kubernetes core components, which run using the Kubernetes hyperkube image, will log ERROR, WARNING and INFO messages. The verbosity of the INFO level log output is controlled by the --v flag, which is set to an integer from 0 to 9. In an RKE CLI or Rancher launched Kubernetes cluster, the --v flag is configured to 2 by default. At this level, the components will log useful steady state information about the service and important log messages that may correlate to significant changes in the system.

In order to troubleshoot an issue, it may be useful to increase the verbosity flag to one of the following:

Update the --v flag in an RKE CLI launched cluster

1. First set the --v flag for the desired components within the cluster.yml. For each of the services you wish to change the verbosity on, you should add an extra_args option with v: "<value>" in the services block, per the example below. The appropriate name for each service within this block can be found within the RKE documentation. N.B. Please see the separate section below for updating the log verbosity of the etcd component

     services:
       kube-api:
         extra_args:
           v: '9'

2. Having set the flag in the cluster.yml, run rke up --config cluster.yml to update the cluster with the new configuration.

Update the --v flag in a Rancher launched cluster

Navigate to the cluster within the Rancher UI and click Edit Cluster, then Edit as YAML. For each of the services you wish to change the verbosity on, you should add an extra_args option with v: "<value>" in the services block of the cluster, per the example below.

N.B. Please see the separate section below for updating the log verbosity of the etcd component.

  services:
    kube-api:
      extra_args:
        v: '9'

The appropriate name for each service within this block can be found within the RKE documentation.

Having set the verbosity flag, click Save at the bottom of the page, to update the cluster.

etcd

The etcd component is configured to log at an INFO level by default, in an RKE CLI or Rancher launched Kubernetes cluster, but this can be set to DEBUG level by setting the --debug=true flag.

Update etcd verbosity in an RKE CLI launched cluster

1. First set the --debug=true flag, within the cluster.yml cluster configuration file, under extra_args for the etcd service, per the following example:

     services:
       etcd:
         extra_args:
           debug: 'true'

2. Having set the flag in the cluster.yml, run rke up --config cluster.yml to update the cluster with the new configuration.

Update etcd verbosity in a Rancher launched cluster

Navigate to the cluster within the Rancher UI and click Edit Cluster, then Edit as YAML. Set the --debug=true flag under extra_args, for the etcd service, per the following example:

  services:
    etcd:
      extra_args:
        debug: 'true'

Having set the debug flag, click Save at the bottom of the page, to update the cluster.