After reboot some OSDs are reported as being down.

This document (7021460) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Enterprise Storage 4

Situation

After rebooting a storage node, "ceph osd tree" output shows not all the OSDs (Object Storage Daemons) belonging to that node are up:

-2 1.33096     host ses-node-X
1 0.21329         osd.1           down 1.00000          1.00000
4 0.21329         osd.4           up 1.00000          1.00000
7 0.90439         osd.7           down 1.00000          1.00000

Resolution

A possible workaround is to execute the following command:

ceph-disk activate-all

Cause

With a larger number of OSDs per host, it can happen that the ceph-disk timeout is reached resulting in not all OSDs being activated after a server reboot.

Additional Information

To verify this is the case, the journalctl output for affected OSD devices (in the below example excerpt output for the device sdi2) will show the following:

:~ # journalctl -u ceph-disk@dev-sdi2.service
-- Logs begin at di 2017-07-11 14:06:49 CEST, end at wo 2017-07-19
21:17:45 CEST. --
jul 11 14:07:09 sesnode-4 systemd[1]: Stopped Ceph disk activation: /dev/sdi2.
jul 11 14:07:09 sesnode-4 systemd[1]: Starting Ceph disk activation: /dev/sdi2...
...
jul 11 14:09:10 sesnode-4 systemd[1]: ceph-disk@dev-sdi2.service: Main process exited, code=exited, status=124/n/a
jul 11 14:09:10 sesnode-4 systemd[1]: Failed to start Ceph disk activation: /dev/sdi2.
jul 11 14:09:10 sesnode-4 systemd[1]: ceph-disk@dev-sdi2.service: Unit entered failed state.
jul 11 14:09:10 sesnode-4 systemd[1]: ceph-disk@dev-sdi2.service: Failed with result 'exit-code'.

The "code=exited, status=124" in the above output indicates the ceph-disk timeout was reached.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.