Pulling images in Rancher with security scanning software fails with error "failed to unmount target: device or resource busy"

This document (000021683) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Rancher 2.10.x with RKE2 clusters.

Situation

In the Rancher enviroment, there could be a situation where pulling images from the Rancher registry fails with mount error mentioned in the example below:

/var/lib/rancher/rke2/agent/containerd/tmpmounts/containerd-mount3127911380: failed to unmount target /var/lib/rancher/rke2/agent/containerd/tmpmounts/containerd-mount3127911380: device or resource busy: unknown"

Resolution

Review the nodes to confirm if there is any security scanning software running on them.
Disable the scanning software on the nodes or exclude paths from scans mentioned in the error.

Cause

When /var/lib/rancher/rke2/agent/containerd/tmpmounts/containerd-* is created and mounted, scanning software starts scanning it, but doesn't finish before containerd times out on the EBUSY signals.
This is a known behavior as mentioned here https://github.com/containerd/containerd/issues/5538

Status

Top Issue

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.