Running DeepSea stage 3 again results in additional OSD's (Object Storage Daemons) on existing OSD devices

This document (7024008) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Enterprise Storage 6

OSD Encryption

Situation

Re-running DeepSea stage 3 or running the disks deployment runner again results in additional OSDs being created on any already existing OSD devices.

Resolution

NOTE: The issue described here is resolved with deepsea-0.9.33 and ceph-14.2.13.450+g65ea1b614d-lp151.2.28.1 or later.

If the cluster is not running at least the above versions then refrain from running DeepSea stage 3 multiple times or manually running the disk deployment runner.

A possible workaround can be to adjust the "/srv/salt/ceph/configuration/files/drive_groups.yml" file in such a way that no devices would match, or that only specific new devices will be matched. It can then be verified by looking at the output of "salt-run disks.list" on the master node that no devices are being matched e.g. for each node the return should show no device information or only specific new device information:

<osd_node>.<fqdn>:
      ----------
      data_devices:
      db_devices:
      wal_devices:

Also see the "Additional Information" section below.

Cause

Due to a bug in ceph-volume, device mapper devices created when encryption is used are always marked as available for OSD deployment.

Additional Information

Looking at the output of "ceph-volume inventory" or "ceph-volume inventory --format jason-pretty" shows the device mapper devices and their availability as True, for example:

/dev/mapper/gARFUj-qlA9-PQHB-qqps-HkBG-l11C-q19Ezo 1.07 TB True True

In addition to the above mentioned workaround, it should also be possible to run certain components of stage 3 "standalone" or to create a custom stage 3 that excludes disk deployment.

Alternatively contact SUSE Technical Support for assistance.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.