btrfs-cleaner high CPU utilization and severe performance issues

This document (000020696) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 15 Service Pack 2 and later
SUSE Linux Enterprise Server 12 Service Pack 3 and later

Situation

Server is suffering severe performance issues.

Looking at 'top' output shows btrfs-cleaner using very high percentage of CPU cycles.

There may be evidence of hangs in btrfs_qgroup_rescan_worker threads reported in messages log.

Trying to manipulate files on one or more btrfs filesystems may hang or takes a long time to complete (e.g. renaming a file).

Resolution

An available work-around is to identify the btrfs filesystem in distress (in some cases it may be the root filesystem) and disable quotas on that filesystem using the following syntax:

#  btrfs quota disable <filesystem>

e.g. to disable quotas on the root filesystem:

#  btrfs quota disable /

To verify quota is disabled for a given filesystem, use:

# btrfs qgroup show  <filesystem>

e.g.

# btrfs qgroup show /
ERROR: can't list qgroups: quotas not enabled

Where quotas are required and can not be disabled, customers with an appropriate SUSE support entitlement may open a support request in order to receive engineering test code which should address the problem.

Cause

There is a known performance issue that can occur where quotas are enabled and subvolume deletion occurs. For the problem to occur, the deletion has to include the removal and cleanup of a large shared subtree.

This means, if the subvolume being deleted is not a snapshot, it will not cause any problems. If the subvolume has undergone a large amount of changes but has almost no shared subtree with the original one, again there will be no problem. If the subvolume itself is small, then still no problem.

This problem is being discussed in an upstream thread with a view to creating a solution to the problem.

Additional Information

Note that btrfs quotas are enabled by default.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.