How to monitor Canal or Calico in RKE2 Downstream Clusters

• Rancher v2.x.
• A Rancher-managed RKE2 cluster with Calico or Canal CNI.
• Rancher-monitoring installed.

Canal and Calico offer metrics that can be exposed and consumed by rancher-monitoring. This article shows how to do it.
 

Prerequisite for Calico

Calico's RKE2 addon does not export Prometheus metrics by default.

To enable this option in the rke2-calico addon, you can go to Cluster Management > Select the cluster > Edit Config > Add-on: Calico. And add the following options in the corresponding "felixConfiguration" and "installation" sections:

felixConfiguration:
  prometheusMetricsEnabled: true
installation:
  typhaMetricsPort: 9093

If your cluster's CNI is Canal, it already exports Prometheus metrics by default, as the following option is enabled in the rke2-canal addon:

    calico:
      felixPrometheusMetricsEnabled: true

After this prerequisite is fulfilled, there are two steps involved to gather these metrics in Prometheus:

Create the Services 

• Canal:

Only one service is needed to be able to extract the metrics from the canal pods through the 9091 metrics port:

apiVersion: v1
kind: Service
metadata:
  name: calico-felix-metrics
  namespace: kube-system
  labels:
    k8s-app: calico-felix
spec:
  clusterIP: None
  ports:
  - port: 9091
    protocol: TCP 
    name: metrics-port
  selector:
    k8s-app: canal

• Calico:

Two services are needed. One to export the calico-felix metrics and another one for the calico-typha metrics. The headless service to export metrics from the calico-kube-controllers is there by default so there is no need to create it.

apiVersion: v1
kind: Service
metadata:
  labels:
    k8s-app: calico-felix
  name: calico-felix-metrics
  namespace: calico-system
spec:
  clusterIP: None
  ports:
  - name: metrics-port
    port: 9091
  selector:
    k8s-app: calico-node

apiVersion: v1
kind: Service
metadata:
  labels:
    k8s-app: calico-typha
  name: calico-typha-metrics-headless-svc
  namespace: calico-system
spec:
  clusterIP: None
  ports:
  - name: metrics-port
    port: 9093
  selector:
    k8s-app: calico-typha

After the service YAMLs are applied, you can proceed to the next step:

Create the ServiceMonitors

• Canal:

  apiVersion: monitoring.coreos.com/v1
  kind: ServiceMonitor
  metadata:
    name: rancher-monitoring-canal
    namespace: kube-system
  spec:
    selector:
      matchLabels:
        k8s-app: calico-felix
    endpoints:
    - port: metrics-port
      relabelings:
      - sourceLabels:
        - __meta_kubernetes_endpoint_node_name
        targetLabel: instance

• Calico:

   

  apiVersion: monitoring.coreos.com/v1
  kind: ServiceMonitor
  metadata:
    name: rancher-monitoring-calico
    namespace: calico-system
  spec:
    endpoints:
    - port: metrics-port
      relabelings:
      - sourceLabels:
        - __meta_kubernetes_endpoint_node_name
        targetLabel: instance
    selector:
      matchExpressions:
      - key: k8s-app
        operator: In
        values: [ "calico-felix", "calico-typha","calico-kube-controllers" ]

After both the Service and ServiceMonitors are applied, the target will appear in Prometheus, and the metrics will be scraped. Be aware that it might take a minute for Prometheus to see the new target and scrape the metrics.