HANA cluster failures due to full /tmp filesystem

The primary node in the HANA cluster experienced a resource failure, resulting in an instance stop and subsequent failover to the secondary node. Analysis of the logs from the HANA cluster resource agents showed that all HANA_CALL operations failed, leading to the HANA instance being marked with a status of HANA_STATE_DEFECT, as indicated by the logs:

SAPHanaTopology(rsc_SAPHanaTopology_HDB00)[280472]: INFO: ACT: hdbnsutil not answering - using global.ini as fallback - srmode=
SAPHanaTopology(rsc_SAPHanaTopology_HDB00)[280472]: ERROR: ACT: check_for_primary:  we didn't expect srmode to be: DUMP: <00000000  0a  |.|#01200000001>
SAPHanaTopology(rsc_SAPHanaTopology_HDB00)[280472]: WARNING: ACT: sht_monitor_clone: HANA_STATE_DEFECT (primary/secondary state could not be detected at this point of time)

The cluster responded effectively by stopping the HANA instance on the primary node and successfully failing over to the secondary node, which restored the functionality of the HANA cluster. However, after this failover process, the original primary instance node was unable to start. Further investigation into this issue revealed that the /tmp filesystem on the primary node was completely full, with 100% usage recorded at the time of the incident. 

The '/tmp' filesystem plays a critical role in the operation of both the HANA instance and the overall cluster stack. In scenarios where issues arise due to this filesystem being full, resolving them typically involves either cleaning up or expanding the /tmp filesystem. Once free space is available again in /tmp, the HANA cluster can be restarted and is expected to resume normal operations.

To address the issue of '/tmp' filesystem usage, a maintenance update has been released for HANA cluster resource agents for both Scale-Up and Scale-Out configurations. This fix avoids the use of the '/tmp' filesystem, ensuring that the SAP HANA resource agents remain operational even when '/tmp' is full. It is recommended to update the SAPHanaSR or SAPHanaSR-ScaleOut package to the latest version, or at least to the versions specified below: