NFS-Ganesha with haproxy does not fail over from active node

This document (000021227) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Enterprise Storage 7.1

Situation

NFS-Ganesha in a Ceph Orchestrator based HA setup does not fail over

Resolution

1. The fail-over will only happen when the NFS-Ganesha node disappears

Ceph orchestrator's HA implementation is limited to specifically cover the case when a node completely disappears, either by being shut down or by losing its network connection. Stopping the NFS-Ganesha service on a node will just cause the ceph orchestrator to try to restart it. Removing the systemd unit file(s) will cause it to redeploy NFS Ganesha on the same node (unless the service is set to "unmanaged", but in that case nothing will happen at all...)

Also, it will not fail-over a hanging NFS-Ganesha daemon. This scenario is usually caused by a busy or locked-up MDS. Moving the NFS gateway of the cluster to a different node will not resolve the situation in this case.

2. Insufficient amount of potential NFS-Ganesha nodes

The NFS HA setup needs at least three nodes where NFS-Ganesha can run. Further more, if there are N nodes the service is allowed to be deployed, it can run on at most N-1 nodes at the same time for fail-over to work. For example, if three nodes node1, node2 and node3 may run NFS-Ganesha, the placement specification should look like this in this example (note the "count" parameter):

service_type: ingress
service_id: nfs.nfs-ha
service_name: ingress.nfs.nfs-ha
placement:
  count: 2 # <-- one node less than hosts listed
  hosts:
  - node1
  - node2
  - node3
spec:
  port: 12049

Cause

The failure scenario is not covered by ceph's orchestrator or the the service specification does not allow fail-over.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.