Cluster mount succeeds but resource shows as failed (OPENAIS, OCFS2, CLUSTER)

This document (7004243) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 11

Situation

After setting up a Filesystem resource with a monitor operation on an openais cluster the resource shows an erronous behaviour. In any monitor tool, be it the GUI or from commandline the Filesystem resource shows as failed. Checking on the commandline with

   mount

shows that the resource was in fact mounted.

   /dev/dm-0 on /var/lib/xen/images type ocfs2 (rw,_netdev,cluster_stack=pcmk)

In the /var/log/messages you could see a sequence similar to

   start of Filesystem resource

RA output: (fs:1:start:stderr) 2009/08/18_14:10:53 INFO: Running start for /dev/disk/by-id/scsi-36005076b074b2b7f4a7d5f5700000009 on /etc/xen/images

   Filesystem resource start succeeded

crmd: [9935]: info: process_lrm_event: LRM operation fs:1_start_0 (call=11, rc=0, cib-update=20, confirmed=true) complete ok

   Monitor of Filesystem resource failure

crmd: [9935]: info: process_lrm_event: LRM operation fs:1_monitor_20000 (call=12, rc=-2, cib-update=0,
confirmed=true) Cancelled unknown exec error

It is also viable to see errors like that the Filesystem is mounted and that dmesg should be checked

lrmd: [14624]: info: RA output: (fs:1:start:stderr) Device or resource busy while mounting /dev/dm-0 on /etc/xen/images. Check 'dmesg' for more information on this error.

this is the case if you did a cleanup on the resource and is secondary to the above mentioned error.

Resolution

The reason for this behaviour lies in the directory attribute that was selected for this Filesystem resource and the fact that it is actually only a symlink to another directory. Assuming that the symlink is from

ls -ld /etc/xen/images
lrwxrwxrwx 1 root root 12 Aug 15 03:53 /etc/xen/images -> /var/lib/xen/images

the directory attribute selected was

/etc/xen/vm/images

while the cluster mounts the device to the directory the Operating system follows the symlink, so in /etc/mtab is

/dev/some_device /var/lib/xen/images

Then after the mount, that succeeds and can be seen in the log files, the cluster does a monitor check which is configured by default. The monitor now checks /etc/mtab for the entry from the directory attribute, which is in this example /etc/xen/vm/images. But the Operating System followed the symlink so in /etc/mtab there is only /var/lib/xen/images. So the monitor fails.

mtab entry

/dev/dm-0 /var/lib/xen/images ocfs2 rw,_netdev,cluster_stack=pcmk 0 0

This can result in the node being fenced.

The solution to this is not to use any symlink in the Filesystem attributes.

Additional Information

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.