How to enable cluster resource tracing

This document (7022678) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise High Availability Extension 15
SUSE Linux Enterprise High Availability Extension 12
SUSE Linux Enterprise High Availability Extension 11 SP4
SUSE Linux Enterprise High Availability Extension 11 SP3

Situation

Resource tracing can help to understand why a resource operation is failing. When enabled it will provide some debug output files.
Often the OCF return codes are not enough to fully understand the failure.
Enabling resource tracing on the failing operation is often useful and provides debug output from the resource agent script.
Resource tracing is currently available for Resource Agents written in bash. For Resource Agents written in Python the feature does not exist at the moment.

Resolution

Enable resource tracing:

crm resource trace <resource name> <operation>
              <resource name> - any defined primitive in the cluster
              <operation>  - start, stop, monitor, probe

Disable resource tracing:

crm resource untrace <resource name><operation>

Debug trace file location:

/var/lib/heartbeat/trace_ra/<resource>/*timestamp

Notes:

If no <operation> is given, it will attempt to trace all operations for the provided resource.
If tracing the monitor operation, number of trace files can grow quickly and should be monitored and cleaned up appropriately and periodically if tracing for extended periods.
Adds the attribute "trace_ra=1" to the operation in the primitive definition.
Example: op monitor enabled="true" interval="20" timeout="45" trace_ra="1"
crm resource tracing does not work for Stonith resources. Stonith resources scripts are located in the following directory "/usr/lib64/stonith/plugins/external/" and various methods can be used to debug these various types of scripts (bash, perl, python). For bash scripts you can modify the script and use "set -x" to enable logging of every command executed and will be logged to SYSLOG and "set +x" to disable logging.
Resource tracing is only supported by OCF Resource Agents.

Enabling resource tracing for operations other than the monitor operation will trigger a resource restart. Pacemaker will stop and start the resource in order to activate tracing. Same applies to the untrace command.
Enabling or disabling tracing of the monitor operation will take affect on the next running of the monitor operation without restarting the resource.

Additional Information

OCF return codes

Additional info: crm resource help trace

Location of resource scripts: /usr/lib/ocf/resource.d/*

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.