Preventing a Fence Race in Split Brain (COROSYNC,PACEMAKER)

This document (7022467) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise High Availability Extension 12

Situation

During a Split Brain in a two node cluster the situation might occur where both nodes fence each other. This is referred to as "Fence Race". An example in the logs of two nodes might look like

Node1 sees Node2 gone and fences

Nov 20 15:17:40 [117052] node1        cib:   notice: crm_update_peer_state_iter:    Node node2 state is now lost | nodeid=168364360 previous=member source=crm_update_peer_proc
Nov 20 15:17:41 [117056] node1    pengine: warning: pe_fence_node:    Node node2 will be fenced because the node is no longer part of the cluster

Node2 sees Node1 gone and fences at the same time

Nov 20 15:17:40 [16727] node2        cib:   notice: crm_update_peer_state_iter:    Node node1 state is now lost | nodeid=168364359 previous=member source=crm_update_peer_proc
Nov 20 15:17:41 [16731] node2    pengine: warning: stage6:    Scheduling Node node1 for STONITH

the resulting effect is, that both nodes fence each other. While Data Integrity is maintained this results in a complete loss of all services.

Resolution

The solution for this is to add to one fencing device in the cluster configuration the parameter

   pcmk_delay_max=<Seconds>

which, in case of an IPMI Device could look like

primitive brie_stonith_ducal stonith:external/ipmi \
        params pcmk_delay_max=20 hostname=ducal ipaddr=10.162.192.209 userid=admin passwd=xxxx interface=lanplus \
        op monitor interval=1800 timeout=20

this will make it more likely, that one fencing device will have a delay. It is at that moment irrelevant which node fences which node, as there is no way for a Cluster without Quorum to determine the right node to be fenced.

This Parameter can be applied to any Fencing Device. IPMI above is only an example.

This still does not ensure to 100% that no fence race will take place. There can still be the situation that one node has an inherent time advantage but has a bigger random delay, so both nodes meet in the middle and kill each other.

With SLES 12 SP2 there is another parameter introduced

pcmk_delay_base=<Seconds>

description below, that makes it possible in some scenarios to ensure that one node survives.

Additional Information

With SLES 12 SP2 there is the introduction of

pcmk_delay_base

as a parameter. In a setup with 2 different fence devices this can be used to ensure that in case of a split brain one specific node survives.

one primitive would be not delayed

primitive fencing-1 stonith:fence_ipmilan \
params pcmk_delay_base=0 ...

while the other one gets the wait base as the value of consensus from the file

/etc/corosync/corosync.conf

if it would be for example

consensus: 36000

then the other fence would read

primitive fencing-2 stonith:fence_ipmilan \
params pcmk_delay_base=36 ...

without compromising the general functionality of fencing.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.