Infrastructure pod /etc/resolv.conf different than kubelet resolv.conf

This document (000020897) is provided subject to the disclaimer at the end of this document.

Environment

RKE1 on an OS running systemd-resolved

Situation

An infrastrucutre pods (one that relies on the node network to communicate rather than the kubernetes overlay network) has a different resolv.conf than kubelet or the node on which it is being hosted.

Resolution

There are multiple options to correct this issue:

Update the symlink for /etc/resolvd.conf from /run/systemd/resolve/stub-resolv.conf to /run/systemd/resolve/resolv.conf
Confirm that /run/systemd/resolve/resolv.conf has the correct DNS entries in it
Disable systemd-resolved if it is no longer needed

After making any of these changes, you will need to restart the impacted pods

Cause

systemd-resolved is an updated method of managing DNS lookups on modern Linux operating systems. By default, /etc/resolv.conf is symlinked to /run/systemd/resolve/stub-resolv.conf that points only to the loopback DNS stub of 127.0.0.1. Changes to /etc/resolv.conf only modify /run/systemd/resolve/stub-resolv.conf. Because this is a stub meant to lookup entries against the configured nameservers in /run/systemd/resolve/resolv.conf, some system tools that rely on networking outside of the host, such as Docker, are configured to check if systemd-resolved is running and use /run/systemd/resolve/resolv.conf directly if it is. This can lead to a discrepancy between what the majority of the host is using and what Docker containers are using to resolve DNS queries.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.