osd's reported as not scrubbed immediately after mon #2 was upgraded to SES6

This document (000019621) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Enterprise Storage 6
SUSE Enterprise Storage 5

Situation

osd's reported as not scrubbed immediately after mon #2 was upgraded to SUSE Enterprise Storage 6.

salt:~ # ceph health detail
HEALTH_WARN 6244 pgs not deep-scrubbed in time; 3307 pgs not scrubbed in time
PG_NOT_DEEP_SCRUBBED 6244 pgs not deep-scrubbed in time
    pg 9.1fff not deep-scrubbed since 2019-12-12 23:44:45.356836
    pg 9.1ffd not deep-scrubbed since 2019-12-18 02:27:07.121590
    pg 9.1ffc not deep-scrubbed since 2020-01-11 03:48:38.158690
    pg 9.1ffb not deep-scrubbed since 2019-12-23 00:46:11.665995
    pg 9.1ffa not deep-scrubbed since 2020-01-16 03:13:00.194279
    pg 9.1ff9 not deep-scrubbed since 2020-01-09 01:57:21.680976
    pg 9.1ff8 not deep-scrubbed since 2020-01-09 01:55:42.461517
    pg 9.1ff6 not deep-scrubbed since 2019-12-05 22:41:57.335234
    pg 9.1ff5 not deep-scrubbed since 2020-01-19 08:22:29.397862
    pg 9.1ff3 not deep-scrubbed since 2020-01-03 00:12:21.638803
    pg 9.1ff2 not deep-scrubbed since 2019-12-21 01:16:40.461801
    pg 9.1ff1 not deep-scrubbed since 2019-12-10 20:40:25.485020
    pg 9.1ff0 not deep-scrubbed since 2019-12-29 17:51:20.050806
    pg 9.1fef not deep-scrubbed since 2020-01-11 03:13:02.814670
    pg 9.1fee not deep-scrubbed since 2020-01-02 23:57:35.081450
    pg 9.1fec not deep-scrubbed since 2019-12-12 23:16:58.629363
    pg 9.1fea not deep-scrubbed since 2020-01-11 03:10:19.442291
    pg 9.1fe8 not deep-scrubbed since 2019-12-19 14:11:32.018310
    pg 9.1fe7 not deep-scrubbed since 2019-12-23 00:36:50.900879
    pg 9.1fe6 not deep-scrubbed since 2019-12-19 00:16:29.522252
    pg 9.1fe5 not deep-scrubbed since 2020-01-16 03:07:17.328692
    pg 9.1fe3 not deep-scrubbed since 2019-12-17 05:41:33.439822
    pg 9.1fe2 not deep-scrubbed since 2019-12-05 21:22:14.491569
    pg 9.1fe1 not deep-scrubbed since 2020-01-14 17:55:28.638895
    pg 9.1fe0 not deep-scrubbed since 2019-12-10 20:35:13.925199
    pg 9.1fdf not deep-scrubbed since 2019-12-26 05:54:37.386976
    pg 9.1fdd not deep-scrubbed since 2019-12-07 21:11:35.742477
    pg 9.1fdc not deep-scrubbed since 2020-01-09 01:48:17.673544
    pg 9.1fdb not deep-scrubbed since 2019-12-06 19:20:57.125293
    pg 9.1fda not deep-scrubbed since 2020-01-06 04:06:16.339087
    pg 9.1fd7 not deep-scrubbed since 2020-01-09 01:41:53.198900
    pg 9.1fd6 not deep-scrubbed since 2019-12-28 18:46:31.394207
    pg 9.1fd5 not deep-scrubbed since 2020-01-12 13:51:31.865562
    pg 9.1fd4 not deep-scrubbed since 2020-01-12 13:50:40.164491
    pg 9.1fd3 not deep-scrubbed since 2019-12-10 20:22:22.706058
    pg 9.1fd1 not deep-scrubbed since 2020-01-02 23:40:19.079870
    pg 9.1fd0 not deep-scrubbed since 2020-01-02 23:26:35.089949
    pg 9.1fcf not deep-scrubbed since 2019-12-05 21:30:21.554120
    pg 9.1fce not deep-scrubbed since 2019-12-07 21:12:08.593495
    pg 9.1fcc not deep-scrubbed since 2019-12-10 19:51:50.166980
    pg 9.1fca not deep-scrubbed since 2019-12-12 23:12:23.913156
    pg 9.1fc9 not deep-scrubbed since 2019-12-29 17:48:23.833781
    pg 9.1fc8 not deep-scrubbed since 2019-12-10 19:50:27.373999
    pg 9.1fc7 not deep-scrubbed since 2019-12-05 21:04:48.028653
    pg 9.1fc6 not deep-scrubbed since 2020-01-18 00:06:12.850531
    pg 9.1fc5 not deep-scrubbed since 2020-01-19 08:12:37.157970
    pg 9.1fc4 not deep-scrubbed since 2020-01-12 13:18:21.454455
    pg 9.1fc3 not deep-scrubbed since 2019-12-20 17:32:19.056567
    pg 9.1fc2 not deep-scrubbed since 2019-12-26 05:22:54.682190
    pg 9.1fc1 not deep-scrubbed since 2019-12-28 17:34:23.837827
    6194 more pgs...
PG_NOT_SCRUBBED 3307 pgs not scrubbed in time
    pg 9.1ffd not scrubbed since 2020-01-15 15:18:58.358539
    pg 9.1ffa not scrubbed since 2020-01-16 03:13:00.194279
    pg 9.1ff7 not scrubbed since 2020-01-21 02:01:49.967096
    pg 9.1ff6 not scrubbed since 2020-01-15 15:08:26.959554
    pg 9.1ff5 not scrubbed since 2020-01-19 08:22:29.397862
    pg 9.1ff3 not scrubbed since 2020-01-17 17:57:43.354134
    pg 9.1ff2 not scrubbed since 2020-01-19 00:48:17.165405
    pg 9.1fed not scrubbed since 2020-01-20 21:31:51.375437
    pg 9.1fe8 not scrubbed since 2020-01-16 16:01:39.467409
    pg 9.1fe6 not scrubbed since 2020-01-16 16:01:38.073323
    pg 9.1fe5 not scrubbed since 2020-01-16 03:07:17.328692
    pg 9.1fe2 not scrubbed since 2020-01-15 15:01:46.681528
    pg 9.1fe1 not scrubbed since 2020-01-14 17:55:28.638895
    pg 9.1fdd not scrubbed since 2020-01-17 17:57:42.271212
    pg 9.1fdb not scrubbed since 2020-01-16 15:56:43.588628
    pg 9.1fda not scrubbed since 2020-01-20 21:34:32.756608
    pg 9.1fd1 not scrubbed since 2020-01-16 16:00:00.433421
    pg 9.1fd0 not scrubbed since 2020-01-16 15:51:41.223135
    pg 9.1fcf not scrubbed since 2020-01-15 15:01:47.333718
    pg 9.1fce not scrubbed since 2020-01-17 17:56:52.004187
    pg 9.1fc7 not scrubbed since 2020-01-15 14:56:43.556745
    pg 9.1fc6 not scrubbed since 2020-01-18 00:06:12.850531
    pg 9.1fc5 not scrubbed since 2020-01-19 08:12:37.157970
    pg 9.1fc3 not scrubbed since 2020-01-17 17:54:47.415088
    pg 9.1fbf not scrubbed since 2020-01-16 03:06:25.050079
    pg 9.1fbe not scrubbed since 2020-01-16 15:46:44.587189
    pg 9.1fbb not scrubbed since 2020-01-20 21:27:42.317465
    pg 9.1fb9 not scrubbed since 2020-01-14 19:18:51.484441
    pg 9.1fb8 not scrubbed since 2020-01-16 15:46:36.836487
    pg 9.1fb5 not scrubbed since 2020-01-19 00:25:18.615322
    pg 9.1fb4 not scrubbed since 2020-01-20 21:09:08.047828
    pg 9.1fb3 not scrubbed since 2020-01-17 17:54:42.990449
    pg 9.1fb1 not scrubbed since 2020-01-20 20:58:06.825964
    pg 9.1fae not scrubbed since 2020-01-20 21:27:52.325779
    pg 9.1fac not scrubbed since 2020-01-16 15:41:45.422057
    pg 9.1fa7 not scrubbed since 2020-01-20 21:24:44.596217
    pg 9.1fa6 not scrubbed since 2020-01-19 00:12:41.566821
    pg 9.1fa5 not scrubbed since 2020-01-14 19:18:43.944557
    pg 9.1fa3 not scrubbed since 2020-01-17 17:55:25.425735
    pg 9.1f9f not scrubbed since 2020-01-20 21:24:44.549135
    pg 9.1f9e not scrubbed since 2020-01-16 15:41:39.180575
    pg 9.1f9a not scrubbed since 2020-01-15 14:53:20.641303
    pg 9.1f99 not scrubbed since 2020-01-14 17:54:11.759034
    pg 9.1f97 not scrubbed since 2020-01-16 03:01:54.251109
    pg 9.1f96 not scrubbed since 2020-01-14 19:15:50.492337
    pg 9.1f93 not scrubbed since 2020-01-19 00:09:42.445792
    pg 9.1f91 not scrubbed since 2020-01-19 00:35:21.332252
    pg 9.1f90 not scrubbed since 2020-01-15 14:46:44.170937
    pg 9.1f8f not scrubbed since 2020-01-17 17:48:42.488453
    pg 9.1f88 not scrubbed since 2020-01-16 15:36:41.024000
    3257 more pgs...

Resolution

The messages :about "pgs not scrubbed in time" and "pgs not deep-scrubbed in time" messages went away when stage.3 in step 8 of section 6.15 of the deployment guide.
https://documentation.suse.com/ses/6/single-html/ses-deployment/#upgrade-main-policy 

In addition the setting was moved to the [global] heading, using "Adjusting ceph.conf with Custom Settings"
https://documentation.suse.com/ses/6/single-html/ses-admin/#ds-custom-cephconf

[global]
osd_scrub_threshold = 0.25
osd_scrub_during_recovery = false
osd_scrub_sleep = 0.2
osd_max_scrubs = 1
osd_scrub_interval_randomize_Ratio = 0.5
osd_scrub_chunk_min = 5
osd_scrub_chunk_max = 20
osd_scrub_begin_hour = 19
osd_scrub_end_hour = 6
osd_scrub_min_interval = 1209600
osd_scrub_max_interval  = 3628800
osd_deep_scrub_interval = 4838400

Cause

The following settings osd_scrub_max_interval and osd_deep_scrub_interval are used by both OSDs and MONs.

The OSDs use them to determine when to run scrub, and the MONs use them to check if they need to show the warning. When they had their overrides in '[osd]' section, only OSDs were using them, so running scrub less frequently than by default, while the MONs still used the default settings.

On SES5 the MONs did not complain because by default the warnings were disabled: "mon_warn_not_deep_scrubbed": "0", "mon_warn_not_scrubbed": "0"
In SES6 the warnings are enabled, so after the upgrade they started to be observed.

The solution to move the scrub setting to [global] section is correct.
We could also just move `osd_scrub_max_interval`, `osd_deep_scrub_interval` to [global] leaving other options in [osd], but it might be more confusing.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.