After upgrading to SES 6 DeepSea stages fail with DNS / FQDN errors

This document (000019935) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Enterprise Storage 6 (SES 6)

Situation

The cluster was upgraded from SES 5.5 to SES 6, after this when running DeepSea stage 2, an error similar to the following is seen:

Exception: According to the 'domain' grain, the cluster does not have a DNS domain configured

When running stage 3, an error similar to the following is seen:

fqdn : ['fqdn deepsea_node.mydns.name does not match minion id deepsea_node']
domain : ["\nIn order to determine a `domain` which is required for some features in DeepSea you need to set-up the FQDN properly.\nExample:\nA device with the hostname `myhost` in the parent domain example.com has the fully qualified domain name myhost.example.com.

Resolution

Properly configure Salt and DeepSea to use FQDN's (Fully Qualified Domain Names).

Cause

With SES 5 using FQDN's were not required, but with SES 6 onwards this is a requirement.

Additional Information

To adjust Salt and DeepSea to use FQDN's take the following steps;

A. Salt:

1) Make a backup of everything under "/srv/pillar/ceph/proposals/".
2) Run `salt '*' pillar.items` and save the output somewhere for reference.
3) Change the hostname to FQDN on all nodes and verify after that `hostname -f` properly returns the expected FQDN.
4) Update the minion id in "/etc/salt/minion_id", to do this for all nodes at the same time, from the admin node run for example:

salt -I cluster:ceph cmd.run 'hostname -f > /etc/salt/minion_id'

5) Restart the salt-minion service on all nodes using `systemctl restart salt-minion.service`. This can be done for all minions by again from the admin node using for example:

salt -I cluster:ceph cmd.run 'systemctl restart salt-minion.service'

6) Rename the keys in "/etc/salt/pki/master/minions/" directory to match the new minion ids.

B. Adjust DeepSea:

1) Remove any existing '*.sls' files under "/srv/pillar/ceph/proposals", for example:

find /srv/pillar/ceph/proposals/ -name '*.sls' -exec rm '{}' ';'

2) Run DeepSea's stage 1 to regenerate all the SLS files with the correct minion IDs, before there should have been for example "/srv/pillar/ceph/proposals/role-mgr/cluster/ses-node01.sls" and other similarly named files. After there should be "/srv/pillar/ceph/proposals/role-mgr/cluster/ses-node01.mydns.name.sls" etc.
3) Edit "/srv/pillar/ceph/proposals/policy.cfg" and make sure the various lines are going to match the FQDN's correctly.
4) Run DeepSea stage 2 to update the pillar data.
5) Finally run `salt '*' pillar.items` again, and compare this output against the pillar data from step A. 2. The information should be the same, except for shortname->FQDN changes.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.