DeepSea stage 2 and 3 returns object "has no attribute" errors

This document (7021092) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Enterprise Storage 4

Situation

Running DeepSea stage 2 and or 3 again to apply configuration changes results in the following errors being returned:

Running Stage 2:

:~ # salt-run state.orch ceph.stage.2
...
[ERROR   ] Run failed on minions: ses_nodeXX.dns_name
Failures:
    ses_nodeXX.dns_name:
        Data failed to compile:
    ----------
        Rendering SLS 'base:ceph.admin.key.default' failed: Jinja variable 'salt.utils.templates.AliasedLoader object' has no attribute 'keyring.secret'

ses_nodeXX.dns_name:
Name: push.proposal - Function: salt.runner - Result: Changed Started: - 11:59:13.283646 Duration: 375.968 ms
Name: refresh_pillar1 - Function: salt.state - Result: Changed Started: - 11:59:13.660015 Duration: 715.502 ms
Name: configure.cluster - Function: salt.runner - Result: Changed Started: - 11:59:14.376005 Duration: 1038.914 ms
Name: refresh_pillar2 - Function: salt.state - Result: Changed Started: - 11:59:15.415411 Duration: 679.597 ms
----------
          ID: admin key
    Function: salt.state
      Result: False
     Comment: Run failed on minions: ses_nodeXX.dns_name
              Failures:
                 ses_nodeXX.dns_name:
                      Data failed to compile:
                  ----------
                      Rendering SLS 'base:ceph.admin.key.default' failed: Jinja variable 'salt.utils.templates.AliasedLoader object' has no attribute 'keyring.secret'
     Started: 11:59:16.095164
    Duration: 876.353 ms
     Changes:

Running Stage 3:

:~ # salt-run state.orch ceph.stage.3
...
[ERROR   ] Run failed on minions: ses_nodeXX.dns_name, ses_nodeXX.dns_name, ses_nodeXX.dns_name, ses_nodeXX.dns_name, ...
Failures:
    ses_nodeXX.dns_name:
    ----------
              ID: /etc/ceph/ceph.conf
        Function: file.managed
          Result: False
         Comment: Unable to manage file: Jinja variable 'salt.utils.templates.AliasedLoader object' has no attribute 'rgw.configurations'
         Started: 11:56:55.822120
        Duration: 341.568 ms
         Changes:

[ERROR   ] Run failed on minions: ses_nodeXX.dns_name
Failures:
    ses_nodeXX.dns_name:
        Data failed to compile:
    ----------
        Rendering SLS 'base:ceph.osd.auth.default' failed: Jinja variable 'salt.utils.templates.AliasedLoader object' has no attribute 'keyring.file'

Resolution

Execute the following command to again distribute the SUSE Enterprise Storage (SES) modules to the minions:

salt '*' saltutil.sync_modules

Cause

The SES salt modules normally deployed when running DeepSea stage 0 are missing from the minion nodes, either because the files were accidentally deleted or if "salt '*' saltutil.clear_cache" were executed from the admin node.

Additional Information

To verify this is the case, check if any files are present on the minions in the following directories:

/var/cache/salt/minion/extmods/modules/
/var/cache/salt/minion/files/base/_modules/

The directory listing for these directories should return something like the following:

:~ # ll /var/cache/salt/minion/extmods/modules/
total 40
-rw------- 1 root root 748 Jul 13 12:19 advise.py
-rw------- 1 root root 2603 Jul 13 12:19 cephdisks.py
-rw------- 1 root root 1152 Jul 13 12:19 keyring.py
-rw------- 1 root root 1699 Jul 13 12:19 mon.py
-rw------- 1 root root 7133 Jul 13 12:19 osd.py
-rw------- 1 root root 1111 Jul 13 12:19 retry.py
-rw------- 1 root root 517 Jul 13 12:19 rgw.py
-rw------- 1 root root 2881 Jul 13 12:19 wait.py
-rw------- 1 root root 1125 Jul 13 12:19 zypper_locks.py

When running the "salt '*' saltutil.sync_modules" command successfully it will also list all the modules being copied, showing the following output for each affected minion:

:~ # salt '*' saltutil.sync_modules
ses_nodeXX.dns_name:
    - modules.advise
    - modules.cephdisks
    - modules.keyring
    - modules.mon
    - modules.osd
    - modules.retry
    - modules.rgw
    - modules.wait
    - modules.zypper_locks

If the modules already exists for a minion or minions, only the minion names are listed:

:~ # salt '*' saltutil.sync_modules
ses_nodeXX.dns_name:
ses_nodeXX.dns_name:
ses_nodeXX.dns_name:
ses_nodeXX.dns_name:

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.