drbd online verify stalls (DRBD OPENAIS)

This document (7009306) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 11
SUSE Linux Enterprise Server 11 Service Pack 1

Situation

With drbd version 8.2.4 there is online verification introduced. Meaning the data on a drbd device can be verified online and in production. The drbd user guide explains how to do this, http://www.drbd.org/users-guide/s-use-online-verify.html
On systems with a very high performance, especially if coupled with a high sync rate this online verify can actually result in a stalled verify. This is characterized by checkin /proc/drbd and getting a result similar to

version: 8.3.11 (api:88/proto:86-96)
GIT-hash: 0de839cee13a4160eed6037c4bddd066645e23c5 build by phil@fat-tyre,
2011-06-29 11:37:11
0: cs:VerifyS ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
    ns:5232620 nr:0 dw:2224 dr:5357005 al:43 bm:319 lo:2 pe:174462 ua:8192 ap:1
ep:1 wo:d oos:0
        [>....................] verified: 1.9% (5016/5104)M
        finish: 2:54:50 speed: 488 (488) want: 307,200 K/sec

where the values for pe (pending) and ua (unacknowledged) rise accompanied by entries in /var/log/messages like

Aug 26 12:22:51 mercury kernel: [ 2599.550987] block drbd0: [drbd0_worker/3155] sock_sendmsg time expired, ko = 4294966875

Resolution

The solution for this issue is to adjust the net and syncer settings for drbd. These are set in the global section /etc/drbd.conf or the appropriate include file.

net {
sndbuf-size 10M;
rcvbuf-size 10M;
ping-int 2;
ping-timeout 2;
connect-int 2;
timeout 5;
ko-count 5;
max-buffers 128k;
max-epoch-size 8192;
}

where max-buffers is increased to 128k, the maximum. And sndbuf-size and rcvbuf-size are drastically increased. The value has to be determined by testing, 4M could be considered a starting point.

syncer {
    c-plan-ahead 20;
    c-min-rate 1M;
    c-max-rate 300M;
    c-fill-target 2M;
   verify-alg md5;
   al-extents 3389;
}

then in syncer the static rate is replaced by dynamical c-plan-ahead, c-min-rate, c-max-rate. and c-fill-target. Where c-min-rate is the minimum syncer rate, c-max-rate the old static value and c-fill-target could be half of the above, in the net section set, sndbuf-size and rcvbuf-size. As with these this value is best determined by testing, a starting point could be 2M.

The dynamical sync rate is available since drbd version 8.3.10

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.