Slow or incomplete path initialization during server startup

This document (7007530) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 10 Service Pack 3

Situation

During server startup, the passive paths to LUNs on a SAN were not initializing properly. This could be seen by monitoring the boot process, or viewing the output of `lsscsi` which would appear as follows:

# /usr/bin/lsscsi -g
[0:0:0:0]    cd/dvd  TEAC     DVD-ROM DV28SV   D.0L  /dev/sr0  /dev/sg0
[2:0:32:0]   enclosu DP       BACKPLANE        1.07  -         /dev/sg1
[2:2:0:0]    disk    DELL     PERC 6/i         1.22  /dev/sda  /dev/sg2
[3:0:0:0]    disk    SUN      CSM100_R_FC      0612  -         /dev/sg3
[3:0:0:1]    disk    SUN      CSM100_R_FC      0612  -         /dev/sg4
[3:0:0:2]    disk    SUN      CSM100_R_FC      0612  -         /dev/sg5
[3:0:0:3]    disk    SUN      CSM100_R_FC      0612  -         /dev/sg6
[3:0:0:4]    disk    SUN      CSM100_R_FC      0612  -         /dev/sg7
[3:0:0:5]    disk    SUN      CSM100_R_FC      0612  -         /dev/sg8
[3:0:0:6]    disk    SUN      CSM100_R_FC      0612  -         /dev/sg9
[3:0:0:7]    disk    SUN      CSM100_R_FC      0612  -         /dev/sg10
[3:0:0:8]    disk    SUN      CSM100_R_FC      0612  -         /dev/sg11
[3:0:0:9]    disk    SUN      CSM100_R_FC      0612  -         /dev/sg12
[4:0:0:0]    disk    SUN      CSM100_R_FC      0612  /dev/sdb  /dev/sg13
[4:0:0:1]    disk    SUN      CSM100_R_FC      0612  /dev/sdc  /dev/sg14
[4:0:0:2]    disk    SUN      CSM100_R_FC      0612  /dev/sdd  /dev/sg15
[4:0:0:3]    disk    SUN      CSM100_R_FC      0612  /dev/sde  /dev/sg16
[4:0:0:4]    disk    SUN      CSM100_R_FC      0612  /dev/sdf  /dev/sg17
[4:0:0:5]    disk    SUN      CSM100_R_FC      0612  /dev/sdg  /dev/sg18
[4:0:0:6]    disk    SUN      CSM100_R_FC      0612  /dev/sdh  /dev/sg19
[4:0:0:7]    disk    SUN      CSM100_R_FC      0612  /dev/sdi  /dev/sg20
[4:0:0:8]    disk    SUN      CSM100_R_FC      0612  /dev/sdj  /dev/sg21
[4:0:0:9]    disk    SUN      CSM100_R_FC      0612  /dev/sdk  /dev/sg22

In the above output, all devices on host3 are missing their corresponding /dev/sd* device. This causes further problems down the line in such things as multipath:

3600a0b8000199e1500003cf545ebf11f dm-3 SUN,CSM100_R_FC
[size=100G][features=1 queue_if_no_path][hwhandler=1 rdac]
\_ round-robin 0 [prio=4][enabled]
 \_ 4:0:0:8 sdj 8:144 [active][ghost]
\_ round-robin 0 [prio=3][active]
 \_ 3:0:0:8 sdt 65:48 [active][ready]
3600a0b8000199e1500003cf045e2d9a5 dm-10 SUN,CSM100_R_FC
[size=20M][features=1 queue_if_no_path][hwhandler=1 rdac]
\_ round-robin 0 [prio=3][active]
 \_ 4:0:0:1 sdc 8:32  [active][ready]
3600a0b8000199e1500003cf345e410c3 dm-5 SUN,CSM100_R_FC
[size=100G][features=1 queue_if_no_path][hwhandler=1 rdac]
\_ round-robin 0 [prio=4][enabled]
 \_ 4:0:0:6 sdh 8:112 [active][ghost]
\_ round-robin 0 [prio=3][active]
 \_ 3:0:0:6 sdr 65:16 [active][ready]

The middle MPIO device in the above output is missing the second priority group containing the passive path - which was not available during the creation of the multipath maps.

The SAN encountering this problem was a SUN CSM100_R_FC. However, there is a possibility of seeing this problem in other SAN environments as well.

Resolution

In SLES10, udev uses a hardcoded timeout of 3 seconds while waiting for sysfs device files to be created. In this case, the SAN took longer than 3 seconds to fully initialize paths on the second scsi host, so the device files were not being created in time for multipath to add them to the maps.

To resolve this issue, an updated udev package (dated newer than December of 2010) must be installed and the following setting must be added to /etc/udev.conf:

udev_sysfs_timeout="10"

With this setting in place, the timeout is increased to 10 seconds and all paths are initialized in time for MPIO.

After making this change, rebuild the initrd, using `mkinitrd`, to ensure the updated udev configuration file is available during the initrd phase of the boot process.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.