"ceph-salt update" fails with ceph-salt.update formula not found

This document (000020623) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Enterprise Storage 7.x

Situation

When executing the command "ceph-salt update" on the admin node to update software packages on all nodes, the following failure is seen:

Syncing minions with the master...
Checking if minions respond to ping...
Pinging 5 minions...
Checking if ceph-salt.update formula is available...
Formula not found. Do you want to restart 'salt-master' to load 'ceph-salt.update' formula? (y/n): [n]: y
Restarting 'salt-master' to load 'ceph-salt.update' formula...
Could not find ceph-salt.update formula. Please check if ceph-salt-formula package is installed

There can be various reasons for encountering the above error, to get more information, details of the actual problem can be found in the salt master and/or minion log files. Running the following command from the admin node may also provide additional information:

salt -G ceph-salt:member state.show_sls ceph-salt

Three known causes are:

1. Minion name discrepancies:

2022-03-23 10:25:14,574 [salt.utils.templates:273 ][ERROR   ][12332] Rendering exception occurred
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/salt/utils/templates.py", line 498, in render_jinja_tmpl
    output = template.render(**decoded_context)
File "/usr/lib/python3.6/site-packages/jinja2/asyncsupport.py", line 76, in render
    return original_render(self, *args, **kwargs)
File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
File "/usr/lib/python3.6/site-packages/jinja2/_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
File "<template>", line 2, in top-level template code
File "/srv/pillar/ceph-salt-top.sls", line 2, in top-level template code
    {% set ceph_salt_minions = ceph_salt.get('ceph-salt', {}).get('minions', {}).get('all', []) %}
jinja2.exceptions.UndefinedError: 'None' has no attribute 'get'

2. The "cephadm" package is not installed on all the minions:

2022-04-13 06:56:45,542 [salt.utils.templates:273 ][ERROR   ][16886] Rendering exception occurred
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/salt/utils/templates.py", line 498, in render_jinja_tmpl
    output = template.render(**decoded_context)
File "/usr/lib/python3.6/site-packages/jinja2/asyncsupport.py", line 76, in render
    return original_render(self, *args, **kwargs)
File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
File "/usr/lib/python3.6/site-packages/jinja2/_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
File "<template>", line 62, in top-level template code
jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'home'

3. Stray salt-minion processes on some of the minions:

2022-05-03 13:55:27,407 [salt.utils.templates:273 ][ERROR   ][1482199] Rendering exception occurred
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/salt/utils/templates.py", line 498, in render_jinja_tmpl
    output = template.render(**decoded_context)
File "/usr/lib/python3.6/site-packages/jinja2/asyncsupport.py", line 76, in render
    return original_render(self, *args, **kwargs)
File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
File "/usr/lib/python3.6/site-packages/jinja2/_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
File "<template>", line 1, in top-level template code
jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'id'

Resolution

For cause 1. change the minion names stored in the ceph-salt configuration to match the actual salt minion names as they are registered with the salt master.

For cause 2. verify that the "cephadm" package is installed on all the nodes.

For cause 3. kill any stray "salt-minion" processes and restart the salt-minion services.

Cause

The minions were initially registered to the salt master using short names (non DNS) and added to the ceph-salt configuration. After the minions were added to the ceph-salt configuration, the minions were re-registered to the salt master using DNS names.

The "cephadm" package is not installed on the minion(s).

Stray salt-minion processes on the minion(s).

Additional Information

Cause 1

To see the current minions in the ceph-salt configuration:

ceph-salt config ls ceph_cluster/minions

To list the minions as they are registered with the salt master:

salt-run manage.list_state

To adjust the minion names in the ceph-salt configuration, first export the current configuration using for example:

ceph-salt export > ceph-salt-config_`date +"%Y-%m-%d_%H"`.json

Modify the minion names to match the salt master registered names, then import the adjusted file:

ceph-salt import ceph-salt-config_<ins_date>.json

Cause 2

To verify the cephadm RPM is installed, from the admin node run:

salt '*' cmd.shell "rpm -qa | grep cephadm"

Cause 3

To verify if there are stray salt-minion processes, stop the salt-minion service on the minion(s):

systemctl stop salt-minion.service

Then list any remaining salt-minion processes and kill any found:

ps aux | grep salt-minion
pkill salt-minion

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.