iSCSI Gateway fails to start due to a timeout error when reading the "gateway.conf" object

This document (000019731) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Enterprise Storage 6

Situation

When the iSCSI Gateway service "rbd-target-api.service" is starting up, it encounters a timeout error when reading the "gateway.conf" object. The following type of error messages are captured in "/var/log/rbd-target-api/rbd-target-api.log" of the iSCSI Node:

  DEBUG [common.py:139:_open_ioctx()] - (_open_ioctx) Opening connection to iscsi-images pool
  DEBUG [common.py:146:_open_ioctx()] - (_open_ioctx) connection opened
  DEBUG [common.py:139:_open_ioctx()] - (_open_ioctx) Opening connection to iscsi-images pool
  DEBUG [common.py:146:_open_ioctx()] - (_open_ioctx) connection opened
  DEBUG [common.py:442:lock()] - config.lock attempting to acquire lock on gateway.conf
  DEBUG [common.py:450:lock()] - (Config.lock) waiting for excl lock on gateway.conf object
   ...
  DEBUG [common.py:450:lock()] - (Config.lock) waiting for excl lock on gateway.conf object
  ERROR [common.py:458:lock()] - (Config.lock) Timed out (30s) waiting for excl lock on gateway.conf object
  ERROR [rbd-target-api:2935:<module>()] - Timed out (30s) waiting for excl lock on gateway.conf object
  CRITICAL [rbd-target-api:2750:halt()] - Unable to open/read the configuration object

In the above example, the pool name used by the iSCSI gateway is "iscsi-images", note however that the pool name can be different in other iSCSI deployments.

Resolution

The "gateway.conf" object stored in the iSCSI pool is locked, and the lock must be broken by performing the below example steps (replace the "iscsi-images" pool with the appropriate pool name):

1. Get the lock information of the "gateway.conf" object from the iSCSI pool by running on the Admin Node:

# rados -p iscsi-images lock list gateway.conf
{"objname":"gateway.conf","locks":[{"name":"lock"}]}

# rados -p iscsi-images lock info gateway.conf lock
{"name":"lock","type":"exclusive","tag":"","lockers":[{"name":"client.4711","cookie":"config","description":"","expiration":"0.000000","addr":"192.168.202.32:0/21039112"}

2. Stop the iSCSI Gateway service on the iSCSI Node(s):

# systemctl stop rbd-target-api.service

3. Using the lock information of the "gateway.conf" object printed from step 1, break the lock for the "gateway.conf" object on the iSCSI pool:

# rados -p iscsi-images lock break gateway.conf lock client.4711 --lock-cookie config

4. Start the iSCSI Gateway service:

# systemctl start rbd-target-api.service

5. If the iSCSI Gateway can now read the "gateway.conf" object, the "/var/log/rbd-target-api/rbd-target-api.log" should contain for example the following messages:

DEBUG [common.py:139:_open_ioctx()] - (_open_ioctx) Opening connection to iscsi-images pool
DEBUG [common.py:146:_open_ioctx()] - (_open_ioctx) connection opened
DEBUG [common.py:430:init_config()] - (init_config) using pre existing config object
DEBUG [common.py:139:_open_ioctx()] - (_open_ioctx) Opening connection to iscsi-images pool
DEBUG [common.py:146:_open_ioctx()] - (_open_ioctx) connection opened
DEBUG [common.py:118:_read_config_object()] - _read_config_object reading the config object

Cause

The "gateway.conf" object stored on the iSCSI pool is locked.

Status

Reported to Engineering

Additional Information

The pool storing the "gateway.conf" object is configured in the "/etc/ceph/iscsi-gateway.cfg" file located on the designated iSCSI node(s).

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.