Prometheus Exporter not able to scrape filesystem metrics from nodes when SELinux is enabled.

This document (000020747) is provided subject to the disclaimer at the end of this document.

Environment

Applicable for Linux distribution with SELinux enabled.

Situation

Using Prometheus Graphs, when users are trying to query filesystem metrics of nodes, it would not display any metrics due to SELinux being enabled. SELinux policies at OS level rejects these queries due to insufficient privileges.

PromQL Query:

node_filesystem_*

Error on the Prometheus exporter pod:

level=error ts=2022-07-21T16:13:08.502Z caller=collector.go:169 msg="collector failed" name=filesystem duration_seconds=0.000837846 err="open /host/proc/1/mounts: permission denied"

Resolution

This is not an issue with the Prometheus Node Exporter which gets installed by the Rancher Monitoring Chart but rather an issue from the SELinux enabled systems. By following one of the below process you can prove that this is not a Prometheus Node Exporter issue and involve the Linux Admin to fix this issue.

1.) Disable SELinux, and the Prometheus node exporter should be able to start querying the node-level metrics. This approach is NOT recommended and also may not be approved by the Linux admin as it would weaken the security of the nodes.

2.) If SELinux cannot be disabled, you can modify the helm chart and update the Prometheus-node-exporter section with the below seLinuxOption called spc_t (reference below). However, this would give the container super-privileged access which is NOT recommended.

securityContext: 
    seLinuxOptions: 
      type: spc_t

3.) If super-privileged access cannot be provided to the container, ask the SELinux team to create necessary policies for processes to access the files that node-exporter pods are looking for which is the recommended approach.

Cause

We notice this issue with SELinux enabled cluster nodes.

Status

Top Issue

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.