With Cavium/Qlogic NX2 cards, SLES12 SP3 fails to boot on first reboot following installation.

This document (7023013) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 12 Service Pack 3 (SLES 12 SP3)

Cavium/Qlogic NX2 cards (bnx2x) Boot From SAN (BFS)

Situation

Following a successful initial installation of SLES12 SP3 using offload FCoE, during the first system Boot From San (BFS), the system will drop in to the emergency dracut shell.

Resolution

This needs to be fixed in the dracut package, which in turn is required to be installed at installation time (See additional information section).

Work-around :

1. Following the first reboot, in the Dracut emergency shell, run the following command to discover the network interfaces :

fipvlan -a

2. For each network interface where FCoE offload is configured, issue the following command :

fipvlan -c -s <network interface>

3. Review output to verify that the boot devices have appeared, by running the following command :

dmesg

4. Exit the dracut shell by running the following command :

exit

5. The system will now boot normally.

6. Navigate to the following directory :

/usr/lib/dracut/modules.d/95fcoe/

In this directory, replace the fcoe-up.sh and fcoe-genrules.sh files with the content of the files as detailed in the "Additional Information" section.

7. Rebuild the initramfs so that the script changes are incorporated into the initramfs, by running the following command :

dracut -f

Cause

Lack of Cavium/Qlogic FcoE driver support in the dracut package.

Additional Information

Please replace the following two files, with the content as listed between the code snippets.

Also please verify the appropriate file system rights (-rwxr-xr-x) are set to these files.

File name : fcoe-genrules.sh

[code/]

#!/bin/sh

# We use (fcoe_interface or fcoe_mac) and fcoe_dcb as set by parse-fcoe.sh
# If neither mac nor interface are set we don't continue
[ -z "$fcoe_interface" -a -z "$fcoe_mac" ] && return

# Write udev rules
{
    if [ -n "$fcoe_mac" ] ; then
        printf 'ACTION=="add", SUBSYSTEM=="net", ATTR{address}=="%s", RUN+="/sbin/initqueue --onetime --unique --name fcoe-up-$env{INTERFACE} /sbin/fcoe-up $env{INTERFACE} %s %s"\n' "$fcoe_mac" "$fcoe_dcb" "$fcoe_mode"
        printf 'ACTION=="add", SUBSYSTEM=="net", ATTR{address}=="%s", RUN+="/sbin/initqueue --onetime --timeout --unique --name fcoe-timeout-$env{INTERFACE} /sbin/fcoe-up $env{INTERFACE} %s"\n' "$fcoe_mac" "$fcoe_dcb" "$fcoe_mode"
    else
        printf 'ACTION=="add", SUBSYSTEM=="net", NAME=="%s", RUN+="/sbin/initqueue --onetime --unique --name fcoe-up-$env{INTERFACE} /sbin/fcoe-up $env{INTERFACE} %s %s"\n' "$fcoe_interface" "$fcoe_dcb" "$fcoe_mode"
        printf 'ACTION=="add", SUBSYSTEM=="net", NAME=="%s", RUN+="/sbin/initqueue --onetime --timeout --unique --name fcoe-timeout-$env{INTERFACE} /sbin/fcoe-up $env{INTERFACE} %s"\n' "$fcoe_interface" "$fcoe_dcb" "$fcoe_mode"
    fi
} >> /etc/udev/rules.d/92-fcoe.rules

[/code]

File name : fcoe-up.sh

[code/]
#!/bin/sh
#
# We get called like this:
# fcoe-up <network-device> <dcb|nodcb> <fabric|vn2vn>
#
# Note currently only nodcb is supported, the dcb option is reserved for
# future use.

PATH=/usr/sbin:/usr/bin:/sbin:/bin
type getarg >/dev/null 2>&1 || . /lib/dracut-lib.sh
type ip_to_var >/dev/null 2>&1 || . /lib/net-lib.sh

# Huh? Missing arguments ??
[ -z "$1" -o -z "$2" ] && exit 1

netif=$1
dcb=$2
mode=$3
vlan="yes"

iflink=$(cat /sys/class/net/$netif/iflink)
ifindex=$(cat /sys/class/net/$netif/ifindex)
if [ "$iflink" != "$ifindex" ] ; then
    # Skip VLAN devices
    exit 0
fi

ip link set dev $netif up
linkup "$netif"

netdriver=$(readlink -f /sys/class/net/$netif/device/driver)
netdriver=${netdriver##*/}

write_fcoemon_cfg() {
    [ -f /etc/fcoe/cfg-$netif ] && return
    echo FCOE_ENABLE=\"yes\" > /etc/fcoe/cfg-$netif
    if [ "$dcb" = "dcb" ]; then
        echo DCB_REQUIRED=\"yes\" >> /etc/fcoe/cfg-$netif
    else
        echo DCB_REQUIRED=\"no\" >> /etc/fcoe/cfg-$netif
    fi
    if [ "$vlan" = "yes" ]; then
        echo AUTO_VLAN=\"yes\" >> /etc/fcoe/cfg-$netif
    else
        echo AUTO_VLAN=\"no\" >> /etc/fcoe/cfg-$netif
    fi
    if [ "$mode" = "vn2vn" ] ; then
        echo MODE=\"vn2vn\" >> /etc/fcoe/cfg-$netif
    else
        echo MODE=\"fabric\" >> /etc/fcoe/cfg-$netif
    fi
}

if [ "$netdriver" = "bnx2x" ]; then
    # If driver is bnx2x, do not use /sys/module/fcoe/parameters/create but fipvlan
    modprobe 8021q
    udevadm settle --timeout=30
    # Sleep for 13 s to allow dcb negotiation
    sleep 13
    fipvlan "$netif" -c -s
    need_shutdown
    exit
fi
if [ "$dcb" = "dcb" ]; then
    # wait for lldpad to be ready
    i=0
    while [ $i -lt 60 ]; do
        lldptool -p && break
        info "Waiting for lldpad to be ready"
        sleep 1
        i=$(($i+1))
    done

    while [ $i -lt 60 ]; do
        dcbtool sc "$netif" dcb on && break
        info "Retrying to turn dcb on"
        sleep 1
        i=$(($i+1))
    done

    while [ $i -lt 60 ]; do
        dcbtool sc "$netif" pfc e:1 a:1 w:1 && break
        info "Retrying to turn dcb on"
        sleep 1
        i=$(($i+1))
    done

    while [ $i -lt 60 ]; do
        dcbtool sc "$netif" app:fcoe e:1 a:1 w:1 && break
        info "Retrying to turn fcoe on"
        sleep 1
        i=$(($i+1))
    done

    sleep 1
fi
write_fcoemon_cfg
fcoemon --syslog=yes

need_shutdown
[/code]

IMPORTANT: Make sure the files don't get CRLF line terminators when copied or created with a editor on Microsoft Windows. The file have to have the following signature:

wawa:/tmp # file fcoe-up.sh fcoe-genrules.sh
fcoe-up.sh: POSIX shell script, ASCII text executable
fcoe-genrules.sh: POSIX shell script, ASCII text executable

should it read:

fcoe-up.sh: POSIX shell script, ASCII text executable, with CRLF line terminators

convert back to unix style with:

unix2dos fcoe-up.sh

which will remove the carriage return. If in doubt, use the attached archive and copy it's content to

/usr/lib/dracut/modules.d/95fcoe/

Note :

The mentioned work-around here is required until a solution is available from Cavium.

On December 27, 2017, SUSE has released the dracut-044.1-114.17.1.x86_64 update, which already provides certain support for the Cavium/QLogic cards.

When installing from a SLES12 SP3 ISO, an older dracut version will be installed, with insufficient Cavium / Qlogic driver support available. Hence the need to provide these details in a work-around.
When installing from an updated SLES12 SP3 installation repository, an updated dracut version is currently available which already provides some improvements here for the Cavium / Qlogic driver support, and there would for example be no need to replace the fcoe-up.sh and fcoe-genrules.sh file.

Following a successful installation, it is advised to register, and update the system with available patches, to ensure latest security vulnerabilities, generic bug fixes and system stability patches will be installed.

At time of writing, dracut-044.1-114.17.1.x86_64 was released on December 27, 2017, which is available via the update repositories, and which provides the following FCoE bug fixes :

* Fri Dec 15 2017 daniel.molkentin@suse.com
- Switch back to fipvlan for bnx2fc (bsc#1052840)
* adds 0548-95fcoe-Switch-back-to-using-fipvlan-for-bnx2fc.patch
- 95fcoe: Allow bnc2x driver more time to complete DCB negotiation (bsc#1052840)
* adds 0549-fcoe-up-Increase-sleeptime-to-13s.patch
- 95fcoe: add timeout initqueue entries (bsc#1052840)
* adds 0550-95fcoe-add-timeout-initqueue-entries.patch

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.