DeepSea Stage 3 deployment fails for each of the OSD nodes: Data failed to compile.

This document (7018857) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Enterprise Storage 4

Situation

The automatically created server profile YAML files (/srv/pillar/ceph/proposals/profile-*-X/stack/default/ceph/minions/server_fqdn.yml) were modified to specify only specific devices to host the OSD journals. When then executing Stage 3 of the deployment, the stage fails for each of the OSD nodes in the cluster with:

           ID: storage
    Function: salt.state
      Result: False
     Comment: Run failed on minions: ses_osdnode_1, ses_osdnode_2, ses_osdnode_3, ...
              Failures:
                  ses_osdnode_1:
                      Data failed to compile:
                  ----------
                      Rendering SLS 'base:ceph.osd.default' failed: Jinja error: cannot concatenate 'str' and 'dict' objects
                  Traceback (most recent call last):
                    File "/usr/lib/python2.7/site-packages/salt/utils/templates.py", line 368, in render_jinja_tmpl
                      output = template.render(**decoded_context)
                    File "/usr/lib/python2.7/site-packages/jinja2/environment.py", line 989, in render
                      return self.environment.handle_exception(exc_info, True)
                    File "/usr/lib/python2.7/site-packages/jinja2/environment.py", line 754, in handle_exception
                      reraise(exc_type, exc_value, tb)
                    File "<template>", line 6, in top-level template code
                  TypeError: cannot concatenate 'str' and 'dict' objects

                  ; line 6

                  ---

                  include:
                    - .keyring

                  {% for device in salt['pillar.get']('storage:osds') %}
                  {% set dev = salt['cmd.run']('readlink -f ' + device ) %}    <======================
                  prepare {{ device }}:
                    cmd.run:
                      - name: "ceph-disk -v prepare --fs-type xfs --data-dev --journal-dev --cluster {{ salt['pillar.get']('cluster') }} --cluster-uuid {{ salt['pillar.get']('fsid') }} {{ device }}"
                      - unless: "fsck {{ dev }}1"
                      - fire_event: True
                  [...]
                  ---

Resolution

Verify the syntax that was used in the server_fqdn.yml files, this also includes the spaces.

Cause

Incorrect file syntax.

Additional Information

Below is an example with the correct syntax for specifying separate journal devices (eg. hosted on NVMe devices), note that there should be two empty spaces before each of the lines after the "storage:" line:

storage:
data+journals:
- /dev/disk/by-id/scsi-35000_OSD_DISK1: /dev/disk/by-id/nvme-JOURNAL_DISKA
- /dev/disk/by-id/scsi-35000_OSD_DISK2: /dev/disk/by-id/nvme-JOURNAL_DISKA
- /dev/disk/by-id/scsi-35000_OSD_DISK3: /dev/disk/by-id/nvme-JOURNAL_DISKB
- /dev/disk/by-id/scsi-35000_OSD_DISK4: /dev/disk/by-id/nvme-JOURNAL_DISKB
osds: []

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.