Configure the Rancher Backup Operator with AWS IMDSv2

This document (000021246) is provided subject to the disclaimer at the end of this document.

Environment

Rancher > 2.6 on EC2 nodes with the Rancher Backup Operator installed and IMDSv2 enabled.

Situation

The Rancher backup operator queries the instance metadata service (IMDS) on AWS EC2 nodes to authenticate with the AWS S3 API when using the IAM permissions attached to EC2 nodes method.

When the EC2 nodes have the IMDSv2 setting set to required instead of optional , the backup operator is unable to assume the IAM profile attached to the EC2 nodes. This leads to the Rancher backups failing with the error failed to check if s3 bucket [< >] exists, error: 401 Unauthorized displayed in the UI with the backup job stuck Retrying.

Resolution

Set the http-put-response-hop-limit instance metadata option key to a value of 2 or greater on the EC2 instances of the Rancher local cluster. The hop-limit option limits the number of hops that metadata requests can travel across a network and will affect the ability of the rancher backup operator pod to query the instance metadata and in turn, prevent it from assuming the instance IAM profile.

NOTE:
Please configure the AWS CLI with an account that has the appropriate IAM permissions to describe and change EC2 metadata settings before executing these commands.

Query the existing instance metadata options from the AWS CLI

aws ec2 describe-instances \
    --instance-id < > \
    --query 'Reservations[].Instances[].MetadataOptions'

Update the http-put-response-hop-limit from the AWS CLI

aws ec2 modify-instance-metadata-options \
     --instance-id < > \    
     --http-put-response-hop-limit 2 \ # Should be >= 2  
     --http-endpoint enabled

Additional Information

Further Reading:

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.