ceph-salt apply fails with a timeout for "find an admin host"

This document (000020339) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Enterprise Storage 7

Situation

When upgrading from SES 6 to SES 7, when running "ceph-salt apply" this fails with a timeout similar to the following:

"admin.server.dns.name": {
"jid": "20210607134408601930",
"out": "highstate",
"ret": {
"ceph_orch_|-find an admin host_|-find an admin host_|-set_admin_host": {
"__id__": "find an admin host",
"__run_num__": 57,
"__sls__": "ceph-salt.apply.cephconfigure",
"changes": {},
"comment": "Timeout value reached.",
"duration": 1822532.013,
"name": "find an admin host",
"result": false,
"start_time": "13:44:22.466186"

Resolution

On all relevant servers where the "sshd_confg" was adjusted, if needed create, then add the user "cephadm" to one of the "AllowGroups" specified in the "/etc/ssh/sshd_config" file.

Alternatively on each of the relevant servers, if needed first create the group and user "cephadm", add the user to the group and then finally add the "cephadm" group to the list of allowed groups in the "/etc/ssh/sshd_config" file.

NOTE: It will be need to restart the "sshd.service" to implement any changes made to the sshd configuration file.

Cause

The "/etc/ssh/sshd_config" file was adjusted to only specify specific groups to allow access using the "AllowGroups" option. None of the allowed groups included the "cephadm" user.

Additional Information

In the salt minion log file for the admin node the ssh commands will fail with:

Command 'ssh' failed with return code: 255

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.