SES clock skew detected, custom /etc/ntp.conf wanted

SUSE Enterprise Storage 5 (SES)
SUSE Linux Enterprise Server 12 Service Pack 3 (SLES 12 SP3)

The ceph health command reports clock skew detected

The administrator wants to customize /etc/ntp.conf for the ceph cluster

Any changes made to the /etc/ntp.conf get overwritten when when ceph.stage.[2-4] are applied

Make sure time is in sync, especially among the monitor nodes, to resolve the clock skew errors. A custom /etc/ntp.conf may help. To adjust using a custom /etc/ntp.conf, do the following:

1. Make a copy of the NTP salt state and the ntp.conf.j2 template files
# cp /srv/salt/ceph/time/ntp/files/ntp.conf.j2 /srv/salt/ceph/time/ntp/files/custom-ntp.conf.j2
# cp /srv/salt/ceph/time/ntp/default.sls /srv/salt/ceph/time/ntp/custom.sls

2. Modify the custom salt state file to use the custom template file
# vi /srv/salt/ceph/time/ntp/custom.sls

Change

   - source:
       - salt://ceph/time/ntp/files/ntp.conf.j2

to

   - source:
       - salt://ceph/time/ntp/files/custom-ntp.conf.j2

3. Modify the custom NTP template file to suit your needs. In this case the changes will make NTP on all servers reference the same time source, instead of the admin node. It will also add additional time source servers. Make changes needed that will keep your cluster's time in sync.

NOTE: The following is only an example and may not work in your environment.

# vi /srv/salt/ceph/time/ntp/files/custom-ntp.conf.j2

Change

server {{ salt['pillar.get']('time_server') }}  burst iburst

to

server  0.pool.ntp.org  burst  iburst
server  1.pool.ntp.org  burst  iburst
server  2.pool.ntp.org  burst  iburst
server  3.pool.ntp.org  burst  iburst

4. Change the local global.yml file to reference your custom salt state
# vi /srv/pillar/ceph/stack/global.yml

time_ntp: custom

5. Run ceph.stage.2 to verify your changes
# salt-run state.orch ceph.stage.2
# salt '*' pillar.get time_ntp
ses5node108:
    custom
ses5node107:
    custom
<snip/>

If the time_ntp value in global.yml, in this case "custom," shows up on all nodes, then the changes have been correctly made. If not, correct any mistakes made in the previous three steps and try again.

6. Apply your changes to one node to confirm everything is configured as expected. In this case, the changes were tested on the first monitor node.
# salt 'mon*1' state.apply ceph.time
# cat /etc/ntp.conf

Take a look at /etc/ntp.conf and verify that your changes are there. If the changes are correct, then apply the changes to all nodes.

7. Run ceph.stage.3 to apply your changes to all nodes
# salt-run state.orch ceph.stage.3

8. Restart the ntpd daemon on all affected nodes
# salt '*' cmd.run 'systemctl restart ntpd.service'

9. Check that NTP is in sync on all nodes and that each server reflects the changes made.
# salt '*' cmd.run 'ntpq -p'

The default SES behavior will configure all nodes to use the admin node as the NTP time source. The admin node will use it's own time source. By using /srv/pillar/ceph/stack/global.yml, DeepSea updates will only update our shipping files. The variable will still be enforced to use the custom setup.

The cluster is primarily concerned about time being in sync among the monitor nodes and only allows a time drift of 0.5 seconds among them before detecting a clock skew. Fix time sync among the monitor nodes to remove the error.

Time synchronization services NTP performs, may be replaced with other services. If you want to configure SES to always use NTP, add time_init to the global.yml file.
# vi /srv/pillar/ceph/stack/global.yml

time_ntp: custom
time_init: ntp