Potential pitfalls with multi-NIC and cloud-netconfig

Most SUSE images in AWS and Azure come with a system called cloud-netconfig pre-installed. The cloud-netconfig implementation handles automatic configuration of network interfaces in those Public Cloud frameworks. See https://www.suse.com/c/multi-nic-cloud-netconfig-ec2-azure/ for the initial announcement of cloud-netconfig and details on what it does.

Multi-NIC and routing policies

In case a system has more than one network interface, routing of packets not destined for a local network, i.e. one that is directly associated with a network interface, gets a bit more complicated. The DHCP service of the public cloud service provider provides a default route for all interfaces, but there can only be one default route in the general routing table. Additional, by default, the kernel only takes the destination IP address into  account when selecting a route. This means, without routing policies, packets for destinations beyond local networks will always be routed through the default route. Typically the default route is associated with the first NIC. If traffic based on specific source IP addresses is associated with another NIC, without routing policies, this traffic will flow over the default route. A connection cannot be established.

To enable the system to automatically route packets through the network interface associated with their source IP address, cloud-netconfig creates additional, interface specific routing tables and routing policy rules, in case it detects multiple network interfaces. The NIC configuration and routing policies created by cloud-netconfig are based on the network configuration information available from the cloud framework metadata server. As only the metadata information is taken into account, those routing policies can interfere with more complex routing setups. An example is the overlay network setup used in a kubernetes cluster.

In case you experience misrouting, you can disable cloud-netconfig, and instead configure secondary IPv4 addresses (if any) and any specific routing you may require manually.

Disabling cloud-netconfig

Any network interface that has been configured automatically via cloud-netconfig has a configuration file associated with it in the `/etc/sysconfig/network` directory on the instance. The automatically generated configuration files contain the `CLOUD_NETCONFIG_MANAGE=’yes’` key value pair. If the value is set to `”NO”` (or the pair is removed altogether), cloud-netconfig will not handle secondary IPv4 addresses and  routing policies for the associated network interface. You should disable cloud-netconfig for any network interfaces that need to be configured manually. If you want to prevent any routing policies from being created, disable cloud-netconfig on all network interfaces. Restarting the network is required to remove any existing routing policies. Execute the following commands as user ‘root’ to disable cloud-netconfig on all network interfaces:

sed -i -e “/CLOUD_NETCONFIG_MANAGE=.*$/CLOUD_NETCONFIG_MANAGE=’no’/” /etc/sysconfig/network/ifcfg-eth*
systemctl restart wicked.service

Manual configuration

For any secondary network interface attached to the instance, you must create a new ifcfg-ethX file if one does not already exist (e.g. ‘ifcfg-eth1’ for the second interface) in ‘/etc/sysconfig/network’ as described above. The most basic  configuration looks like this:

BOOTPROTO=’dhcp’
STARTMODE=’hotplug’
CLOUD_NETCONFIG_MANAGE=’no’

Additional parameters are available to configure a variety of properties for the network interface, for example setting a static IP address. For more information refer to the file ‘/etc/sysconfig/network/ifcfg.template’ or
run ‘man 5 ifcfg’.

Please note that in a multi-NIC setup, routing policies may be required to enable the kernel to make routing decisions additionally based on source IP addresses, not solely destination addresses. Please refer to the SUSE documentation for network configuration details.