Can't open watchdog device: /dev/watchdog: Device or resource busy

This document (7008216) is provided subject to the disclaimer at the end of this document.

Environment

SLES 11 SP1 - HAE with the latest available patches installed.
STONITH is using external/sbd

Situation

The STONITH device does not appear to be working correctly to fence nodes. Failing nodes are not fenced correctly as they should be with a correctly configured STONITH device.

Searching through the /var/log/messages file the following is seen:

     Mar 15 12:00:56 node1 sbd: [16150]: ERROR: Cannot open watchdog device: /dev/watchdog: Device or resource busy

Because the sbd STONITH agent requires the use of the watchdog device, having another service holding that device open/busy is interfering with sbd from doing its job in the HA environment.

The sbd agent was configured correctly (/etc/sysconfig/sbd was correct, as was the configuration on the actual device).

Searching further in the messages file, the following is found:

     Mar 15 10:46:10 node1 hpasrd[8955]: Starting with poll 1 and timeout 600.
     Mar 15 10:46:10 node1 hpasrd[8955]: Setting the watchdog timer.
     Mar 15 10:46:10 node1 hpasrd[8955]: Using watchdog HP iLO2 HW Watchdog Timer.

The hpasrd daemon is using the watchdog device.

A quick search on this shows that hpasrd is the Hewlett Packard Automatic Server Recovery Daemon.

node1:~ # chkconfig --list | grep hp
hp-asrd                          0:off 1:off 2:on   3:on   4:on   5:on   6:off
hp-health                        0:off 1:off 2:on   3:on   4:on   5:on   6:off
hp-snmp-agents            0:off 1:off 2:on   3:on   4:on   5:on   6:off
hpsmhd                          0:off 1:off 2:off   3:on   4:on   5:on   6:off
hpvca                             0:off 1:off 2:off    3:on   4:on   5:on   6:off
openhpid                       0:off 1:off 2:off    3:off   4:off   5:off   6:off

Resolution

For our specific case, we disabled hp-asrd in chkconfig (chkconfig hp-asrd off). After this, the sbd resource was able to properly connect to /dev/watchdog and that enabled the STONITH device to properly function inside the environment.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.