How to configure an authenticated forward proxy for Alertmanager in rancher-monitoring

• You have the Alertmanager installed on the cluster.
• You need to configure any receiver integration, e.g., Slack, PagerDuty, Opsgenie, etc., where an authenticated proxy is required.

A proxy configuration, with username and password, can be added to the alertmanager configuration, and the alertmanager UI will automatically obfuscate the password in the Status -> Config view. e.g. proxy_url: http://<user>:xxxxx@<URL>:<port> so this is secured from users with UI access. There are two ways you can define this proxy configuration:

• A non-persistent implementation, e.g. to test the proxy configuration, can be performed by editing the "alertmanager-rancher-monitoring-alertmanager" secret in the "cattle-monitoring-system" namespace post-deployment to set the proxy_url including authentication directly in the URL. However, this solution will not be permanent, as the secret could will be overwritten, deleting the changes, if any update is made to the rancher-monitoring chart.
• As a persistent implementation, you can create a copy of this secret (with the proxy_url including the credentials already set) in the cattle-monitoring-system namespace. Then, you can refer to this new secret in the cattle-monitoring chart alertmanager.alertmanagerSpec.configSecret value to work with your receiver integration.

According to the Alertmanager official documentation, the proxy credentials should be facilitated using the following http_config spec: https://prometheus.io/docs/alerting/latest/configuration/#http_config. However, in the latest versions of the software, the "http_config.proxy_from_environment" field has been removed, avoiding the possibility of setting these credentials using the env variables PROXY_PASS and HTTPS_PROXY. Trying to configure the mentioned credentials using those env variables, will throw the following error message:

level=error ts=2024-06-28T14:29:14.657065268Z caller=klog.go:126 component=k8s_client_runtime func=ErrorDepth msg="sync \"cattle-monitoring-system/rancher-monitoring-alertmanager\" failed: provision alertmanager configuration: failed to initialize from secret: yaml: unmarshal errors:\n line 4: field proxy_from_environment not found in type alertmanager.httpClientConfig"

Then, the solution needs to pass by setting the proxy credentials directly on the URL defined in the http_config.