Creating additional images results in random OSDs being reported as down.

This document (7021060) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Enterprise Storage 4

Situation

On a healthy cluster when creating new additional images this results in random OSDs (Object Storage Daemons) being reported as down.

Resolution

Set the following in "/etc/ceph/ceph.conf" and reboot the cluster nodes:

ms_type = async

NOTE: See the additional information section for more details, take caution when making this change.
Alternatively contact SUSE Customer Care by opening a Service Request.

Cause

The default SimpleMessenger used with SUSE Enterprise Storage 4 (which is based on the Ceph Jewel release) is much more resource intensive and can cause problems on larger clusters.

Additional Information

The following is an excerpt of a related OSD failure logged to the relevant OSDs log file (/var/log/ceph/ceph-osd.X) :

0> 2017-05-16 15:42:42.583883 7f7f5f416700 -1 *** Caught signal (Aborted) **
in thread 7f7f5f416700 thread_name:ms_accepter

ceph version 10.2.5-239-g3a6a822 (3a6a822c8125858afaeac7a1ee0d121d063660f0)
1: (()+0x91e682) [0x55696038a682]
2: (()+0x10b10) [0x7f7f779a8b10]
3: (gsignal()+0x37) [0x7f7f7543b8d7]
4: (abort()+0x13a) [0x7f7f7543ccaa]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x265) [0x556960493775]
6: (Thread::create(char const*, unsigned long)+0xba) [0x55696046398a]
7: (SimpleMessenger::add_accept_pipe(int)+0x66) [0x556960458e26]
8: (Accepter::entry()+0x36b) [0x55696052f86b]
9: (()+0x8744) [0x7f7f779a0744]
10: (clone()+0x6d) [0x7f7f754f0d3d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

There are however known issues with the AsyncMessenger, for more information see:

http://tracker.ceph.com/issues/18761

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.