Booting from NVMe-oF over TCP - Known Issues

SUSE Linux Enterprise Server 15 SP5

Booting from NVMe-oF over TCP is supported under the following conditions:
- SUSE Linux Enterprise Server 15 Service Pack 5 or later
- A SAN storage array supporting NVMe-oF over TCP
- A host system with a BIOS that supports booting from NVMe-oF over TCP.

More details are available in the SLES15 SP5 Storage Administration Guide .

Booting from NVMe-oF over TCP is a recent development on both the BIOS and the operating system side.
While high diligence has been employed to make sure the feature works stably, it it is still new in production environments.


Known Issues:

- NetworkManager currently not supported
Booting from NVMe-oF over TCP currently requires using wicked for network management.
While booting with NetworkManager might work, NetworkManager will not be aware of the special nature of the nbft$X network interfaces. It may try to reconfigure them or shut them down, which is dangerous for network connections that are required for accessing the root file system.
wicked implements the logic to deal with nbft$X interfaces correctly.

Support for NetworkManager will be added in a later SLE 15 Service Pack.


- Different gateways on multiple network interfaces
Routing into disjoint subnets through different gateways via multiple network interfaces is not supported for NVMe-oF over TCP boot.
In such configurations, it can happen that some NVME target subsystems are not reachable by the operating system on the host. This is currently a restriction, caused by different basic routing concepts between the UEFI pre-boot environment and the Linux operating system.

To avoid this problem, make sure that all NVMe subsystems that are required for booting are either connected to the host on local network segments (no routing/gateway required), or are reachable through a single default gateway.


- Boot failure if NVMe fails to connect for more than 10 minutes
If during boot the connection to the NVMeoF server is disrupted, and the server can't be reached for 10 minutes or more, the system will give up trying to connect, and booting will fail, even if the connection is reestablished some time later.

This is fixed in the dracut maintenance update, dracut-055+suse.364.g4c1d0276-150500.3.3.1 released June 2023.


- Multipath issues
In a multipath setup (several host network interfaces used for connecting to the same NVMe-oF subsystem), booting may hang if one interface is down during boot.

This problem has been fixed in the early maintenance updates dracut, nvme-cli and wicked released June 2023:
dracut-055+suse.364.g4c1d0276-150500.3.3.1
nvme-cli-2.4+17.gf4cfca93998a
wicked-nbft-0.6.72-150500.3.7.1


- MTU and other link settings
Setting ethernet link attributes such as the Maximum Transmission Unit (MTU) for nbft$X interfaces is currently not supported.

It can be achieved by creating a custom udev rule and including it in the initramfs. Please contact SUSE Support for details.


- Minor issues
The error message NBFT: jq error while processing ".host_name" may be visible during boot.
This message appears if the system host name had not been set in the BIOS NVMe-oF settings.
It is harmless and can be ignored.

A fix for this issue is provided in the maintenance update dracut-055+suse.364.g4c1d0276-150500.3.3.1 released in June 2023.

--

The NVM Express Logo® and the NVM Express ® , NVMe® , NVMe-oF™, and NVMe- MI™ wordmarks are registered or unregistered service marks of the NVM Express organization in the United States and other countries.

1. NVMe-oF support for virtual machines is available with enhanced OVMF BIOS images that can be obtained from the Timberland SIG .
2. nvme-cli in SLE 15 SP5 (GA) supports only the legacy syntax nvme show-nbft, which has been superseded by nvme nbft show later.

A nvme-cli maintenance update in SLE 15 SP5 will support both commands.

Reported to Engineering