System lock up after XFS reporting possible memory allocation deadlock size 4194320 in kmem_alloc (mode:0x2400240)

This document (7023344) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 15
SUSE Linux Enterprise Server 12 Service Pack 3 (SLES 12 SP3)
SUSE Linux Enterprise Server 12 Service Pack 2 (SLES 12 SP2)
SUSE Linux Enterprise Server 12 Service Pack 1 (SLES 12 SP1)

Situation

While observing a high fragmentation on a XFS filesystem, the kernel starts to report messages similar to:

2018-06-25T15:21:03.532830+00:00 kernel: [8235358.733050] XFS: SubmitThread-DA(56017) possible memory allocation deadlock size 4194320 in kmem_alloc (mode:0x2400240)

and after a while locks up.

Resolution

Due to the complexity of the patch, a backport to SLES 12 SP1 was not possible. The course of action for this patch level is to update to either SLES 12 SP2 or SLES 12 SP3 to apply the fix.

If this issue is experienced on a system running SAP HANA please refer to:

https://launchpad.support.sap.com/#/notes/2655238 (login required!)

SLES 15:
https://download.suse.com/patch/finder/#bu=suse&familyId=7261&productId=60008&dateRange=&startDate=&endDate=&priority=&architecture=&keywords=1095344&xf=7261&xp=7261_55388

SLES 12 SP3 fixes:
https://download.suse.com/patch/finder/#bu=suse&familyId=7261&productId=55388&dateRange=&startDate=&endDate=&priority=&architecture=&keywords=1095344

SLES 12 SP2 LTSS fixes:

https://download.suse.com/patch/finder/#bu=suse&familyId=7261&productId=64687&dateRange=&startDate=&endDate=&priority=&architecture=&keywords=1095344&xf=7261&xp=7261_64687

Cause

In memory data structures associated with a fragmented file (e.g. due to suboptimal use of Direct I/O (DIO) get too big. As the number of fragments grow kmem_alloc is unable to allocate more than 4MB contiguous memory for them.

Additional Information

When using DIO all heuristics that XFS employs to optimize its IO operations are bypassed. Thus users need to carefully understand the implications of DIO and care should be taken to avoid excessive file fragmentation (i.e. using falloc to preallocate large contiguous regions of a file). Thus please understand, the patches added to the XFS code only address the hang situation, not the file fragmentation.

To show the fragmentation status of a given filesystem:

xfs_db -c frag -r <dev>

To start a defragmentation:

xfs_fsr -v <dev>

For further details please see "man xfs_fsr" and "man xfs_db"

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.