Node in a DRBD cluster shows disk is in Inconsistent state

This document (000020325) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise High Availability Extension 12
SUSE Linux Enterprise High Availability Extension 15

Situation

Secondary node in a DRBD pacemaker cluster is in inconsistent state:

czdcpv-uxnfs204:~ # drbdadm status Usr_sap_common
Usr_sap_common role:Primary
volume:0 disk:UpToDate
volume:1 disk:UpToDate
volume:2 disk:UpToDate
volume:3 disk:UpToDate
volume:4 disk:UpToDate
volume:5 disk:UpToDate
volume:6 disk:UpToDate
volume:7 disk:UpToDate
volume:8 disk:UpToDate
czdcpv-uxnfs203 connection:StandAlone


czdcpv-uxnfs203:~ # drbdadm status Usr_sap_common
Usr_sap_common role:Secondary
volume:0 disk:Inconsistent
volume:1 disk:Inconsistent
volume:2 disk:Inconsistent
volume:3 disk:Inconsistent
volume:4 disk:Inconsistent
volume:5 disk:Inconsistent
volume:6 disk:Inconsistent
volume:7 disk:Outdated
volume:8 disk:Inconsistent
czdcpv-uxnfs204 connection:Connecting

Resolution

On the survivor (Primary that is in UpToDate state) we need to change the connection status from StandAlone
to Connecting. We can do that with following command:

drbdadm connect Usr_sap_common

next, on the secondary node that is in Inconsistent state run:

drbdadm connect --discard-my-data Usr_sap_common

which will trigger re-syncing the data from primary to secondary.
Once the sync completes, both nodes show status "UpToDate"
Verify with command:

drbdadm status Usr_sap_common

If the sync is not triggered after the procedure above, in that case, on the secondary node that is
in inconsistent state, instead of running "drbdadm connect --discard-my-data Usr_sap_common" run:

drbdadm invalidate Usr_sap_common

which forces DRBD to consider the data on the local backing storage device as out-of-sync.
Therefore DRBD will copy each and every block from its peer, to bring the local storage device back in sync.
Depending on the disks size and connection speed, the sync can take several hours.

Cause

Split brain can cause DRBD inconsistency.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.