Device-mapper does not use all available paths to a SAN

SUSE Linux Enterprise Server 11

Under certain circumstances, Device-Mapper can create device maps using the physical paths to a LUN, rather than the multipath logical path (which contains all possible paths). This problem is only seen when a single Logical Volume (LV) is comprised of multiple Physical Volumes (PVs) residing on multipath (DM-MPIO) devices.

If the device-mapper table is not built with the MPIO paths, path failover will not work properly and I/O will not be load balanced across active paths.

The root of this issue is a race condition between multipath and udev, which is only seen when LVM Logical Volumes (LVs) are created from more than one Physical Volume (PVs), and those Physical Volumes are available through multiple paths on a SAN (accessed using multipath).

The problem has been resolved in multipath-tools package version 0.4.8-40.19.1 and newer. This issue can also be worked around by removing the following udev rule file:

/lib/udev/rules.d/64-lvm2.rules

After removing this file, the server must be rebooted to resolve the problem.

When using LVM in conjunction with MPIO, the LVM filter (defined in /etc/lvm/lvm.conf) should look similar to the following:

    filter = [ "a|/dev/disk/by-id/dm-uuid-.*mpath-.*|", "r/.*/"]

This filter causes LVM to only scan the multipath device nodes found in /dev/disk/by-id, and to ignore the raw paths (/dev/sdX) to each LUN. However, even with the above filter in place, a race condition is causing device-mapper to build the Logical Volume map incorrectly. This can be seen in the output of dmsetup:

# dmsetup ls --tree
testvg-testlv (253:2)                     
 ├─ (8:48)                              
 ├─36006016073c01400b60df1da84bade11 (253:1)
 │  ├─ (8:32)                              
 │  └─ (8:80)                              
 └─ (8:64)

This dmsetup output shows that two out of the three LUNs used for this volume are being accessed through single paths. Under normal conditions, the dmsetup output should appear as follows:

# dmsetup ls --tree
testvg-testlv (253:3)
 ├─36006016073c01400761c473080eade11 (253:2)
 │  ├─ (8:96)
 │  └─ (8:48)
 ├─36006016073c01400b60df1da84bade11 (253:1)
 │  ├─ (8:32)
 │  └─ (8:80)
 └─36006016073c01400b6e6b01d81bade11 (253:0)
    ├─ (8:16)
    └─ (8:64)

If a server is experiencing this problem, dmsetup can be used to identify the problem and confirm the fix has resolved the issue. The iostat utility can also be used to confirm that all active paths are seeing an appropriate amount of IO.