Removal of Ethernet Cable from one port - IP of the other port will become unreachable

SUSE Linux Enterprise Server 15 SP5
SUSE Linux Enterprise Server 15 SP4

On a system where all network interfaces are connected to the same subnet and Wicked is used for network management, the following is noticed during Hot plug/unplug of the Ethernet cable:

a) Removal of Ethernet cable from one port (port1) of the network interface makes the IP of the other port (port2) unreachable.

b) Removal of Ethernet cable from one port (port1), the IP of this port (port1) still remains reachable.
Expectation is that the IP of the port where the cable is removed from, should become unreachable.

Solution1:

Configuration of different DHCPv4 metrics for each interface in ifcfg and sysctl, and mitigation of ARP-Flux.

Wicked will not remove routes on linkdown event.
With the below settings the kernel will not use routes if they are tagged as linkdown.

1. Add DHCLIENT_ROUTE_PRIORITY to each NIC configuration in /etc/sysconfig/network/ and configure different DHCPv4 metrics for each interface.
This allows wicked to set more than one default route.

The configuration of the first interface ifcfg-eth0 should look like:

STARTMODE=auto
BOOTPROTO=dhcp4
DHCLIENT_ROUTE_PRIORITY=100

The configuration of the second interface ifcfg-eth1 should look like:

STARTMODE=auto
BOOTPROTO=dhcp4
DHCLIENT_ROUTE_PRIORITY=200

2. Ignore routes with the linkdown flag.

In /etc/sysctl.d/ create the 50-shared-net.conf file with the below net.ipv4.conf settings:

# Ignore routes which have the linkdown flag
net.ipv4.conf.eth0.ignore_routes_with_linkdown=1
net.ipv4.conf.eth1.ignore_routes_with_linkdown=1

3.  Mitigate ARP-Flux with sysctl.
Because IP addresses are owned by the host, ARP requests for all IP addresses on all interfaces will be answered.
So if an ARP-Request is coming in on one of the interfaces, the host will answer regardless which interface.
This is also known as ARP-Flux.

Add the following settings to the file /etc/sysctl.d/50-shared-net.conf

# Reply only if the target IP address is local address configured and sender IP address is on the same subnet
net.ipv4.conf.eth0.arp_ignore=2
net.ipv4.conf.eth1.arp_ignore=2

# Always use the best local address for this target
net.ipv4.conf.eth0.arp_announce=2
net.ipv4.conf.eth1.arp_announce=2

# Ensure to disable `rp_filter` to avoid packet dropping by route match
net.ipv4.conf.eth0.rp_filter=0
net.ipv4.conf.eth1.rp_filter=0

Load the new configuration with:

# sysctl --load /etc/sysctl.d/50-shared-net.conf
# wicked ifup eth0 eth1


Solution 2:

Network bond.

For better performance it is recommended to use bonding mode.
Create bonding with balance-alb mode which uses both interface at a time (if they are connected).

The /etc/sysconfig/network/ifcfg-bond0 will look like this:

STARTMODE='auto'
BOOTPROTO='dhcp4'
BONDING_MASTER='yes'
BONDING_SLAVE_0='eth0'
BONDING_SLAVE_1='eth1'
BONDING_MODULE_OPTS='mode=balance-alb miimon=100'
ZONE=public

An IP address belongs to the host and not to the interface. When both interfaces are connected to the same subnet there is no assurance which interface will be used to transmit traffic, and the machine will accept traffic for either IP on either interface. Further on a link-down event, unlike an ifdown event, wicked does not update the routing-table.