Node in cluster will not start or join after being fenced.

This document (7022918) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise High Availability Extension 12
SUSE Linux Enterprise High Availability Extension 11 Service Pack 4

Situation

Starting pacemaker or openais service and it fails to start with a dependency error.
systemd[1]: Dependency failed for Pacemaker High Availability Cluster Manager.

Messages seen in SLES12 log after trying to start pacemaker.service and failing to start / join the cluster.
sbd: [34824]: WARN: Found fencing message - aborting start-up. Manual intervention required!
sbd: [34819]: WARN: Servant for pcmk (pid: 34826) has terminated
sbd: [34819]: WARN: Servant for <path to sbd device > (pid: 34824) has terminated
sbd.sh[34808]: sbd failed; please check the logs.
sbd.sh[34808]: SBD failed to start; aborting.
systemd[1]: sbd.service: Control process exited, code=exited status=1

Resolution

With the "start" mode option set in the /etc/sysconfig/sbd, you will need to manually clear the SBD slot before it will join.
Need to find out <SBD_DEVICE> which can be found in /etc/sysconfig/sbd file.
Syntax to be run on node that was previously fenced:
Example: sbd -d <SBD_DEVICE> message LOCAL clear

You may also issue the command from any node in cluster by specifying the node name instead of "LOCAL"
Syntax: sbd -d <DEVICE_NAME> message <NODENAME> clear
Example: sbd -d /dev/sda1 message node1 clear

Once the node slot is cleared, you should be able to start clustering.
SLES114: rcopenais start
SLES12+: systemctl start pacemaker.service

Cause

This is working as designed / configured. This detects a fencing type message in the sbd slot for the node and will not allow the node to join the cluster until it's been manually cleared.

Additional Information

Option for SLES12 /etc/sysconfig/sbd
SBD_STARTMODE=clean

## Default: always Options: always, clean
# Specify the start mode for sbd. Setting this to "clean" will only allow sbd to start if it was not previously fenced.
See the -S option in the man page.

Option for SLES11 /etc/sysconfig/sbd
SBD_OPTS="-W -P -S 1"

# -S 1 - set the start mode < Reference: man sbd >
# If set to one, sbd will only start if the node was previously shutdown cleanly (as indicated by an exit request message in the slot), or if the slot is empty. A reset, crashdump, or power-off request in any slot will halt the start up.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.