The azure-events resource agent, Pacemaker clusters, and you!

New “azure-events-az” resource agent has been developed.  This post to be updated.

So let’s say you are an Azure customer and you want to be in the know about all the wonderful things this fancy-looking azure-events resource agent (RA) can do for you, but you don’t have time to dissect the code to figure out all of its moving parts. You’re not alone. Software Engineers, hackers, and general tech nerds with lots of time on their hands need not worry because they can read the source code[1] on their own time. In a perfect world, the ability or time to do this wouldn’t be so elusive.

The existing Microsoft documentation[2] provides a basic high-level definition of what this azure-events RA is supposed to do. However, it fails to document the Pacemaker resource agent’s functions with any degree of granularity. This makes it more difficult for a system administrator to take a systematic troubleshooting approach when challenged with a crisis brought on by an issue with a production system, or with knowing how they can effectively predict how user-initiated maintenance events[3] will be processed by Pacemaker.  In-depth user documentation for this specialty resource agent has not existed since the time it was written.  That changes today.

This article serves to resolve these documentation gaps by describing each major component of this resource and how the resource agent works interdependently with the platform.

Components –

Azure Instance Metadata Service (IMDS) – Provides an API endpoint for the Azure Cloud to publish upcoming scheduled events, as well as other metadata, for the instance. Scheduled Events may be user-generated, such as a Restart or Redeploy initiated by the operator with one of Azure’s several user interfaces, or platform-generated, such as planned or unplanned host or network maintenance.  The API is accessed over HTTP at 169.254.169.254, a Link-local address which is standardized as the platform communications IP by several major Cloud Service Providers.

Pacemaker – Together with Corosync, the Pacemaker cluster software suite, which comes as part of the High Availability Extension add-on product or with SLES for SAP Applications images, provides the cluster engine to run highly-available services on SUSE and automate service recovery when a failure is detected.  Pacemaker executes the azure-events resource agent script which probes the platform for events at the resource’s monitor interval.

Python – The language in which the resource agent is written. If the resource agent, itself, fails to execute or an error is logged that can’t be explained by an environmental problem, the issue may be a problem with a python library.

Azure Events Resource Agent (RA) script – Located at /usr/lib/ocf/heartbeat/resources.d/azure-events. This script is executed by Pacemaker when the RA is started, stopped, or runs its monitor operation and is responsible for “pulling” Scheduled Events from the platform.

Now with these definitions out of the way, let’s move on to the fun part. Here’s how it all works:

Checking for Queued Events –

The azure-events resource agent is configured as a cloned resource, meaning the same RA runs on every node in the cluster. A global cluster attribute, azure-events_globalPullState, defines whether any one node in the cluster is currently “PULLING” events from the Azure IMDS; only one cluster node should be pulling events from the platform at a time. When none of the cluster nodes are pulling events, the azure-events_globalPullState value should be IDLE. When the resource monitor runs on any node, it checks the value of this cluster attribute. If IDLE, the resource agent switches the attribute value to PULLING while it queries for new events from the IMDS. Once all events have been processed, or if there are no events to process, the resource agent switches the attribute back to IDLE. If the azure-events monitor also runs on a different node while one node is already PULLING events, the RA on that other node will wait for one second and check the attribute again. It will run this check three times and if the azure-events_globalPullState is still set to PULLING after the third check, the monitor operation will complete and any pending events will need to wait until the next monitor run.

Processing Events (Node States) –

There are four node states per the azure-events_curNodeState attribute: AVAILABLE, STOPPING, IN_EVENT, and ON_HOLD. These are specific terms for this resource agent and should not be confused with Pacemaker’s “standby” and “online” states.  As long as a node is not processing an event, its should always be in an AVAILABLE state. When the RA pulls an event which requires clustered services to be stopped, the Pacemaker service on that node is switched from “online” to “standby” and the attr_curNodeState is changed to “STOPPING” until the RA script confirms that all resources on the node have been stopped. Once all services are stopped, the event(s) can begin without impacting running services. If for any reason the event cannot run, the azure-events_curNodeState will be set to “ON_HOLD”, otherwise, the state is changed to “IN_EVENT” and the event is started. Once all events have been processed, the RA switches the curNodeState back to AVAILABLE. This should switch the Pacemaker status from “standby” to “online” and any resources that can be started up on the node per Pacemaker (pengine) logic will be started up again.

But what about reboots?  How is the RA supposed to execute and take its node out of standby when the system starts Pacemaker while the node is *still in* standby?  Doesn’t that mean the azure-events monitor operation wouldn’t run any more on that node?  Also, what happens if a reboot isn’t the last queued event?

These are fair points when thinking in a strictly Pacemaker perspective.  The local azure-events resource agent wouldn’t be able to switch the curNodeState back to AVAILABLE while Pacemaker is in standby, or pull any additional events for that matter.  This is where the function removeOrphanedEvents and handleRemoteEvent are relevant.  Not only does the azure-events RA check its own node state, but it also checks the states of other nodes in the cluster.  If the curNodeState of a remote cluster node is set to IN_EVENT or STOPPING, but also does not have any pending events from the platform, another instance of the RA will be able to switch it back to AVAILABLE/online once it confirms that all platform events were handled.  It can also handle any additional queued events on behalf of the remote node while that node is in standby.

I hope this write-up is helpful and informative, and as always, if you have any issues running SUSE workloads in Azure, be sure to open a Support Request with the Azure support teams.

[1]: https://github.com/ClusterLabs/resource-agents/blob/master/heartbeat/azure-events.in
[2]: https://docs.microsoft.com/en-us/azure/virtual-machines/workloads/sap/high-availability-guide-suse-pacemaker#pacemaker-configuration-for-azure-scheduled-events
[3]: https://docs.microsoft.com/en-us/azure/virtual-machines/linux/scheduled-events#user-initiated-maintenance