How to speed up / slow down deep-scrubbing

This document (000019684) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Enterprise Storage 5.5
SUSE Enterprise Storage 6

Situation

Customer has noticed that deep-scrubbing has fallen behind.

Resolution

If impacting ceph performance is not a concern, the following could be done:
Configure *osd max scrubs* and *osd scrub load threshold* to do more scrubbing work, but be careful as it will have a huge impact on performance.

Get current configuration by running the following command on an OSD node:

# ceph daemon osd.$osdID config show | egrep "osd_max_scrubs|osd_scrub_load"
Example:
# ceph daemon osd.22 config show | egrep "osd_max_scrubs|osd_scrub_load"
"osd_max_scrubs": "1",
"osd_scrub_load_threshold": "0.500000",

Assuming that impacting cluster clients is of no concern.
If system load average is above .5, then increase osd_scrub_load_threshold,
For system load average see: uptime, cat /proc/loadavg, top, sar -q 1 5, w
if load average is above consider increasing "osd scrub load threshold=", but may want to check randomly through out the day.
salt -I roles:storage cmd.shell "sar -q 1 5"
salt -I roles:storage cmd.shell "cat /proc/loadavg"
salt -I roles:storage cmd.shell "uptime"

Otherwise increase osd_max_scrubs:
ceph tell 'osd.*' injectargs --osd_max_scrubs=2 --osd_scrub_load_threshold=3

Validate with:
# ceph daemon $osdID config show | egrep "osd_max_scrubs|osd_scrub_load"
or
# ceph daemon $osdID config diff

Values will vary on each cluster.
Monitor for desired results with "ceph -s", etc....
If need drop back values back down closer to defaults if the commands negatively impact the cluster.

Cause

Cluster may have been backfilling.
Cluster may have client load.

Status

Top Issue

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.