VolumeNotCreated: Volume <volume_guid> did not finish being created even after we waited 187 seconds or 61 attempts. And its status is creating.

This document (7022867) is provided subject to the disclaimer at the end of this document.

Environment

SUSE OpenStack Cloud

Situation

Launching instance from volume with storage backend in Ceph fails with error :

2018-04-16 10:43:23.537 27350 ERROR nova.compute.manager [req-86e4c368-9aae-4661-bdae-4f80873aedb5 305de25c713c79ebf21e6be56f486ffd51c91f8da6e25d8a43efcbdd8c9d2289
9f90be101d8247d58116ffb687189cc6 - - -] [instance: 8053d801-ecb8-48ca-8fcb-22927c18620d] Instance failed block device setup
2018-04-16 10:43:23.537 27350 ERROR nova.compute.manager [instance: 8053d801-ecb8-48ca-8fcb-22927c18620d] Traceback (most recent call last):#012
2018-04-16 10:43:23.537 27350 ERROR nova.compute.manager [instance: 8053d801-ecb8-48ca-8fcb-22927c18620d]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1586, in _prep_block_device#0122018-04-16 10:43:23.537 27350 ERROR nova.compute.manager [instance: 8053d801-ecb8-48ca-8fcb-22927c18620d]     wait_func=self._await_block_device_map_created)#012
2018-04-16 10:43:23.537 27350 ERROR nova.compute.manager [instance: 8053d801-ecb8-48ca-8fcb-22927c18620d]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 514, in attach_block_devices#012
2018-04-16 10:43:23.537 27350 ERROR nova.compute.manager [instance: 8053d801-ecb8-48ca-8fcb-22927c18620d]     map(_log_and_attach, block_device_mapping)#012
2018-04-16 10:43:23.537 27350 ERROR nova.compute.manager [instance: 8053d801-ecb8-48ca-8fcb-22927c18620d]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 512, in _log_and_attach#0122018-04-16 10:43:23.537 27350 ERROR nova.compute.manager [instance: 8053d801-ecb8-48ca-8fcb-22927c18620d]     bdm.attach(*attach_args, **attach_kwargs)#012
2018-04-16 10:43:23.537 27350 ERROR nova.compute.manager [instance: 8053d801-ecb8-48ca-8fcb-22927c18620d]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 404, in attach#0122018-04-16 10:43:23.537 27350 ERROR nova.compute.manager [instance: 8053d801-ecb8-48ca-8fcb-22927c18620d]     self._call_wait_func(context, wait_func, volume_api, vol['id'])#012
2018-04-16 10:43:23.537 27350 ERROR nova.compute.manager [instance: 8053d801-ecb8-48ca-8fcb-22927c18620d]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 363, in _call_wait_func#012
2018-04-16 10:43:23.537 27350 ERROR nova.compute.manager [instance: 8053d801-ecb8-48ca-8fcb-22927c18620d]     {'volume_id': volume_id, 'exc': exc})#012
2018-04-16 10:43:23.537 27350 ERROR nova.compute.manager [instance: 8053d801-ecb8-48ca-8fcb-22927c18620d]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__#012
2018-04-16 10:43:23.537 27350 ERROR nova.compute.manager [instance: 8053d801-ecb8-48ca-8fcb-22927c18620d]     self.force_reraise()#012
2018-04-16 10:43:23.537 27350 ERROR nova.compute.manager [instance: 8053d801-ecb8-48ca-8fcb-22927c18620d]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise#0122018-04-16 10:43:23.537 27350 ERROR nova.compute.manager [instance: 8053d801-ecb8-48ca-8fcb-22927c18620d]     six.reraise(self.type_, self.value, self.tb)#012
2018-04-16 10:43:23.537 27350 ERROR nova.compute.manager [instance: 8053d801-ecb8-48ca-8fcb-22927c18620d]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 353, in _call_wait_func#012
2018-04-16 10:43:23.537 27350 ERROR nova.compute.manager [instance: 8053d801-ecb8-48ca-8fcb-22927c18620d]     wait_func(context, volume_id)#012
2018-04-16 10:43:23.537 27350 ERROR nova.compute.manager [instance: 8053d801-ecb8-48ca-8fcb-22927c18620d]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1258, in _await_block_device_map_created#012
2018-04-16 10:43:23.537 27350 ERROR nova.compute.manager [instance: 8053d801-ecb8-48ca-8fcb-22927c18620d]     volume_status=volume_status)#012
2018-04-16 10:43:23.537 27350 ERROR nova.compute.manager [instance: 8053d801-ecb8-48ca-8fcb-22927c18620d] VolumeNotCreated: Volume 711ae672-1739-4d62-8c01-28ad5b34919c did not finish being created even after we waited 187 seconds or 61 attempts. And its status is creating.

Resolution

Go to crowbar Dashboard -> Barclamps -> OpenStack -> Nova and there click on "Raw" view, which opens up a json config.

Scroll down and look for the following section :

"block_device": {
"allocate_retries": 60,
"allocate_retries_interval": 3
},

The above are defaults, change allocate_retries from 60 to a higher value, for example 300.
Click 'save', and apply, and re-try launching the instance.

Note, these settings will be updated in '/etc/nova/nova.conf.d/100-nova.conf' on controller and compute nodes:

block_device_allocate_retries = 300
block_device_allocate_retries_interval = 3

Cause

Nova expects the volume to be available within defined timeout.
With larger images it might take longer to create the volume than the default configured timeout.

Additional Information

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.