Accessing the Public Cloud Update Infrastructure via a Proxy

SUSE provides public cloud customers with PAYG (Pay-As-You-Go) images on AWS, Azure, and GCP. Instances created from these images connect to a managed update infrastructure. So if you need to update your instances with the latest software updates or install that needed package using zypper, usually you can be assured that the underlying repositories are there with no further hassles. There are exceptions, though.  Instances configured to utilize a proxy server or traverse firewalls, NAT gateways, proxies, security rules, Zscalar, or other security and network devices may run into problems.  The purpose of this post is to address some of the more commonly occurring configuration issues seen with public cloud environments.

Before getting into the issues, what exactly is the update infrastructure? The update infrastructure is a collection of “servers” accessible over the public internet which provide information or software repositories for SLE (SUSE Linux Enterprise) PAYG instances.  These servers are either region servers or update servers.  Region servers, which comprise a small percentage of the infrastructure, are located across the globe for redundancy. These “information” servers exist to provide update server IP addresses depending on the instance’s region.  Update servers make up a majority of the infrastructure and hold the package repositories. At least three update servers serve a given region per cloud provider for high availability and low latency with high data transfer rates.

To properly configure your environment to access the update infrastructure, it’s important to understand what happens during the instance’s initial registration. First, the instance needs to find out what update servers to use.  It does this by contacting a region server over port 443 (HTTPS).  It only needs to successfully query one of the region servers listed in the regionsrv parameter in /etc/regionserverclnt.cfg. During the SSL handshake, the instance communicates with the region server using certificates local to the instance located in /var/lib/regionService/certs. This is when customers typically run into problems with a MITM (man-in-the-middle) proxy, Zscalar, or similar device. These MITM devices intercept the TLS traffic between the instance and the region server and instead utilize a trusted MITM certificate between the proxy and the instance. This MITM certificate is unknown to the region servers. As a result, the connection to the region server is terminated before the instance acquires the required information to continue the registration. Conversely, instances that successfully communicate with the region servers typically have TLS traffic passed through the proxy or device and can acquire the IP addresses for the update servers for the region where the instance is running to continue with the next step.

Assuming the instance successfully communicated with one of the region servers, it now knows the three update server IPs for its region.  The next step in the registration process is to connect to the update servers over port 443 (HTTPS) [1].  The update server that responds first is selected, and a host record added to /etc/hosts resolving smt-<PROVIDER>.susecloud.net [2] to the selected update server’s IP address.  This record is added locally because it will not be publicly resolvable.  This is an important point because some proxy, firewall, or other devices expect to be able to resolve this hostname publicly to an IP address.  After the host record is added,  the instance will import the certificate from the update server into /usr/share/pki/trust/anchors  and then run update-ca-certificates.  This certificate will then be used to create an encrypted session between the instance and the update server.  Once again, this is when customers run into problems with a MITM proxy or similar device.  The certificate from the update servers cannot be intercepted and replaced or the session will fail.

Finally, if all communication passes and there are no unexpected issues, the registration completes with all repositories being made available to the instance.  Be aware that the registration process all depends on the metadata server at 169.254.169.254 being available.  This address should be excluded from being proxied.

What happens if communication fails along the way as described above? The update infrastructure is designed using certificate pinning.  This will cause problems with MITM devices that intercept the pinned certificates and replace them with their own.  The MITM device will need to exempt the update infrastructure IPs from SSL inspection to prevent these problems.  Using free and open-source mitmproxy [3] as an example, this exemption can be performed by adding the update infrastructure IPs to ignore_hosts, which will exclude them from interception.  Consult your device documentation for details on performing this in your environment.  For Zscalar customers, a good place to start is here [4].

Another problem that may be encountered is the resolution of the update server host record [2].  As previously mentioned, this record is only local to the instance and not publicly resolvable.  Therefore, the hostname and IP address added to the hosts file may need to be added to the proxy device. Other device-specific methods may need to be considered, for example, allowing DNS resolution to occur from the client instead of from the proxy.

So how does one know what IP addresses an instance will need to communicate with to successfully register?  For this, we provide the pint tool [5], which is available in the python3-susepubliccloudinfo package.  To get the region server IPs for your public cloud provider, you can run one of the following:

pint google servers --regionserver
pint microsoft servers --regionserver 
pint amazon servers --regionserver

To get the update server IPs for your public cloud provider and region, you can run one of the following (replace the example US East region with the region of instance):

pint google servers --region="us-east4"
pint microsoft servers --region="eastus" 
pint amazon servers --region="us-east-1"

Alternatively without installing the pint tool, this information is also available from a web browser.

GCP: https://susepubliccloudinfo.suse.com/v1/google/servers.json
Azure: https://susepubliccloudinfo.suse.com/v1/microsoft/servers.json
AWS: https://susepubliccloudinfo.suse.com/v1/amazon/servers.json

The log file /var/log/cloudregister can also be consulted to ascertain what IP addresses the instance is attempting to communicate with.

If further assistance is required for PAYG instance registration to the SUSE update infrastructure, open a support case with your cloud provider.

You can also try the repository checker tool that I wrote to automatically correct an instance that cannot access the update infrastructure.  It provides informational output that includes reasons why the instance has issues, including if a proxy is suspected in the cause.  The tool is available at https://github.com/rfparedes/susecloud-repocheck.

[1] As long as the /etc/regionserverclnt.cfg contains parameter httpsOnly = true the client communicates over port 443 (HTTPS). If this parameter is not present or not set to true, communication will occur over port 80 (HTTP).  Update to latest cloud-regionsrv-client for httpsOnly functionality [6].

[2] In AWS, the record is smt-ec2.susecloud.net.  In Azure, the record is smt-azure.susecloud.net.  In GCP, the record is smt-gce.susecloud.net.

[3] https://docs.mitmproxy.org/stable/howto-ignoredomains/

[4] https://help.zscaler.com/zia/certificate-pinning-and-ssl-inspection

[5] https://www.suse.com/c/riddle-me-this/

[6] https://www.suse.com/c/update-infrastructure-access-adios-http/