Cluster fencing not working with SBD partition

This document (7010932) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise High Availability Extension 11 (HAE)
Split Brain Detection (SBD) Partition

Situation

The cluster nodes do not fence properly, usually they don't get fenced when they should. The sbd node list shows:


# /usr/sbin/sbd -d /dev/sdb1 list
0    hn1    clear    

# /usr/sbin/sbd -d /dev/sdc1 list
0    hn1    clear    
1    hn2    clear

The cluster information base (CIB) shows the following node list:


# /usr/sbin/cibadmin -Q
--snip--
<nodes>
  <node id="hn1" type="normal" uname="hn1"/>
  <node id="hn2" type="normal" uname="hn2"/>
</nodes>
--snip--

Resolution

Recreate the SDB partitions and perform a rolling restart of the cluster nodes. For example,

1. Display the current SBD partitions.

# cat /etc/sysconfig/sbd
SBD_DEVICE="/dev/sdb1;/dev/sdc1;/dev/sdd1"
SBD_OPTS="-W"

2. Confirm that the shared device names listed in /etc/sysconfig/sbd exist with the same device path on all nodes, and are the same physical media (ie confirm /dev/sdb1, /dev/sdc1 and /dev/sdd1 exist on all nodes and all nodes have the same device path for each. In other words, one device cannot be /dev/sdc1 on node1 and /dev/sdf1 on node2).

3. Confirm that the same /etc/sysconfig/sbd exists on all nodes. The devices listed must exist on all nodes.

# scp /etc/sysconfig/sbd hn2:/etc/sysconfig/

4. Reformat the SBD partition on each devices listed:

# sbd -d /dev/sdb1 -d /dev/sdc1 -d /dev/sdd1 create

5. Reboot one node in the cluster. When it comes back online, reboot another node. Repeat the process until each node in the cluster has been rebooted.

Cause

The sbd node list did not match the CIB node list.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.