Scheduling deep-scrubbing schedule for off peak hours

SUSE Enterprise Stoarge 5.5
SUSE Enterprise Stoarge 6

Customer needs to schedule deep-scrubbing for off peak hours to help minimize impact on cluster. 

It's possible to set up a deep-scrubbing schedule if there are off peak hours for the deep-scrubbing to occur. 

Default behavior is for all pg's to be deep-scrubbed once a week.  It may be necessary to increase interval values  to every other week to ensure that pg's are deep-scrubbed periodically. 

Example:
Start scrubbing at 8am, go until 17 (5pm), begin_week_day (Monday), end_week_day (Friday), the configuration would be as below.

osd_scrub_begin_hour = 8
osd_scrub_end_hour = 17
osd_scrub_begin_week_day = 1
osd_scrub_end_week_day = 6
osd_scrub_max_interval = 1209600
osd_scrub_min_interval = 259200
osd_deep_scrub_interval = 1209600

Note: “deep scrubbing” process will continue after “osd_scrub_end_hour” parameter until the “deep scrubbing” for the pg is completed.

In some cases the cluster is already busy, so deep-scrubbing does not occur frequent as needed.
See: https://www.suse.com/support/kb/doc/?id=000019684

Also see:
https://www.suse.com/support/kb/doc/?id=000019621

General ceph documentation reference:
https://docs.ceph.com/en/latest/rados/configuration/osd-config-ref/

In some cases, customers choose to use their on scripts to ensure deep-scrubbing occurs on their own schedule. 

Deep-scrubbing can impact client access to the cluster. 

Top Issue

Other information which can help determine deep-scrubbing status

Scrubbing & Deep+Scrubbing Distribution by hours, day of week, or date
Columns 22 & 23 are scrub history
Columns 25 & 26 are for deep-scrub history
These columns will change, if "ceph pg dump" output changes.

# ceph pg dump | head -n 8 | grep "active"
dumped all
1.7f          0                  0        0         0       0     0           0          0    0        0 active+clean 2020-09-16 02:24:25.675595          0'0     1753:247  [31,1,15]         31  [31,1,15]             31          0'0 2020-09-16 02:24:25.675555             0'0 2020-09-16 02:24:25.675555             0

________________________________________
Scrubbing Distribution by Days of the week and hours of the day.
Get Scrubbing distribution by days of the week:
# for date in `ceph pg dump | grep active | awk '{print $22}'`; do date +%A -d $date; done | sort | uniq -c
dumped all
    329 Tuesday
    175 Wednesday
Get Scrubbing distribution per hours of the day:
# for date in `ceph pg dump | grep active | awk '{print $23}'`; do date +%H -d $date; done | sort | uniq -c
dumped all
   23 00
   15 01     
   22 02     
   20 03     
   28 04     
   27 05
  ---[cut here]---

________________________________________
Get Deep+Scrubbing Distribution by Days of the week and hours of the day.
Get Deep+Scrubbing distribution by days of the week:
# for date in `ceph pg dump | grep active | awk '{print $25}'`; do date +%A -d $date; done | sort | uniq -c
dumped all
     51 Friday
     60 Monday
     70 Saturday
     75 Sunday
     45 Thursday
    132 Tuesday
     71 Wednesday

Get Deep+Scrubbing distribution per hours of the day:
# for date in `ceph pg dump | grep active | awk '{print $26}'`; do date +%H -d $date; done | sort | uniq -c
dumped all
     28 00
     20 01
     19 02
     17 03
     24 04
     24 05
---[cut here]---
________________________________________
Get Deep_Scrubbing distribution by date:
# ceph pg dump | grep active | awk '{print $25}' | sort | uniq -c
dumped all
      7 2020-09-08
     25 2020-09-09
     45 2020-09-10
     50 2020-09-11
     70 2020-09-12
     73 2020-09-13
     60 2020-09-14
    125 2020-09-15
     49 2020-09-16

________________________________________
Sort pg by "DEEP_SCRUB_STAMP":

Date & Time:
ceph pg dump | grep active | awk '{print $1,$25,$26}' | sort -k2,1 -k2

Date only:
ceph pg dump | grep active | awk '{print $1,$25}' | sort -k2