'SIGSEGV: segmentation violation' in prometheus container of the prometheus-project-monitoring-0 Pod when enabling Project monitoring on the System Project

Issue

Enabling project monitoring in a Rancher v2.2 cluster, in which cluster monitoring is enabled, fails with the Prometheus Pod in a CrashLoopBackOff.

The prometheus container in the prometheus-project-monitoring StatefulSet fails with an error of the following format:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x669c12]
goroutine 437 [running]:
net/http.(*Client).deadline(0x0, 0xc005381070, 0x40bb8f, 0xc0055e3600)
/usr/local/go/src/net/http/client.go:187 +0x22
net/http.(*Client).do(0x0, 0xc005cdaa00, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:527 +0xab
net/http.(*Client).Do(0x0, 0xc005cdaa00, 0x23, 0xc002802230, 0x9)
/usr/local/go/src/net/http/client.go:509 +0x35
github.com/prometheus/prometheus/scrape.(*targetScraper).scrape(0xc0060fa960, 0x1fd4a60, 0xc00010ec60, 0x1fb2760, 0xc0002eb110, 0x0, 0x0, 0x0, 0x0)
/app/scrape/scrape.go:471 +0x111
github.com/prometheus/prometheus/scrape.(*scrapeLoop).run(0xc00616a100, 0xdf8475800, 0x2540be400, 0x0)
/app/scrape/scrape.go:813 +0x487
created by github.com/prometheus/prometheus/scrape.(*scrapePool).sync
/app/scrape/scrape.go:336 +0x45d

Pre-requisites

• A cluster managed by Rancher v2.2
• Cluster monitoring enabled and Project monitoring enabled on the System project

Resolution

Project monitoring is not compatible with the Rancher System project and should not be enabled in the System project. Starting with Rancher v2.3.0 monitoring of the System project is performed by cluster monitoring, when this is enabled, and the UI prevents enabling of project monitoring on the System project.