How to add additional scrape configs to a Rancher cluster or project monitoring prometheus

Task

The Rancher cluster and project monitoring tools, allow you to monitor cluster components and nodes, as well as workloads and custom metrics from any HTTP or TCP/UDP metrics endpoint that these workloads expose.

This article will detail how to manually define additional scrape configs for either the cluster or project monitoring prometheus instance, where you want to scrape other metrics.

Whether to define the additional scrape config at the cluster or project level would depend on the desired scope for the metrics and possible alerts. If you wish to scope the metrics scraped, and thus possible alerts configured for these metrics, to a project, you could configure the additional scrape config at the project monitoring level. If you wish to scope the metrics at the cluster level, so only those with cluster admin access could see the metrics or configure alerts, you could configure the additional scrape config at the cluster monitoring level.

Pre-requisites

• A Rancher v2.2.x, v2.3.x or v2.4.x managed cluster, with cluster monitoring enabled (and optionally project monitoring enabled, if you wish to configure the additonal scrape config at the project scope).

Resolution

For both cluster and project monitoring the additional scrape config(s) are defined in the Answers section of the Monitoring configuration. This can be found as follows:

• Cluster Monitoring: As a user with permissions to edit cluster monitoring (global admins and cluster owners by default), navigate to the cluster view and click Tools -> Monitoring from the menu bar. Click 'Show advanced options' at the bottom right.
• Project Monitoring: As a user with permissions to edit project monitoring (global admins, cluster owners and project owners by default), navigate to the project and click Tools -> Monitoring from the menu bar. Click 'Show advanced options' at the bottom right.

You can add an array of prometheus.additionalScrapeConfigs in the Answers section here.

For example to define a scrape job of the following:

 - job_name: "prometheus"
   static_configs:
   - targets:
     - "localhost:9090"

You would add the following two definitions to the Answers section:

prometheus.additionalScrapeConfigs[0].job_name = prometheus prometheus.additionalScrapeConfigs[0].static_configs[0].targets[0] = localhost:9090

After adding the answers, click 'Save' and you should now be able to view the target and its status within the Prometheus UI under Status -> Targets.

Further reading

Documentation on the Rancher cluster monitoring can be found here and for Rancher project monitoring here.