How to recover from drbd split brain.

This document (7021212) is provided subject to the disclaimer at the end of this document.

Environment

SUSE OpenStack Cloud 6
SUSE OpenStack Cloud 7

Situation

Output on e.g. secondary node (where in this case postgresql resource xfs-mount is not mounted) :

drbd-overview

0:postgresql/0 Connec/StaAlo Second/Unknow Outdat/DUnkno

1:rabbitmq/0 Connected(2*) Second/Primar UpToDa/UpToDa

Following commands will also show nodes not connected:

On SUSE Cloud6, use : drbdadm cstate
On SUSE Cloud7, use : drbdadm status

Resolution

Disconnect secondary and cleanup :

1. drbdadm disconnect postgresql
2. drbdadm secondary postgresql
3. drbdadm connect --discard-my-data postgresql

On the other node (the split brain survivor), connect to start sync :

drbdadm connect postgresql

Cause

Split brain scenario of drbd resource.

Additional Information

https://docs.linbit.com/doc/users-guide-84/s-resolve-split-brain/

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.