SUSE Support

Here When You Need Us

I/O to LUNs hang / stall under high load when using xen-blkfront

This document (7018590) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 11 Service Pack 4 (SLES 11 SP4)
SUSE Linux Enterprise Server 12 Service Pack 3 (SLES 12 SP3)
SUSE Linux Enterprise Server 12 Service Pack 4 (SLES 12 SP4)

Situation

Under high I/O load, for example when running database stress tests on the Virtual Machines, I/O hangs / stalls are seen.

Resolution

Increase the default "gnttab_max_frames" of "32" to a higher value by starting the Hypervisor (Dom0) with the kernel parameter "gnttab_max_frames=xxx".

Cause

The default setting can be insufficient for high I/O environments.

Additional Information

General recommendation for determining the proper value for "gnttab_max_frames" is to multiply by 2 the number of attached disks. Alternatively the following can be used to determine the appropriate value:
gnttab_max_frames=(6*#NICs + 8*#discs) * 1.5

To adjust the parameter edit the "/boot/grub/menu.lst" file of the Hypervisor (Dom0) and look for a line similar to the below example:

kernel /xen.gz


If the desired value is determined to be 256, adjust the line as follows and save the file:
kernel /xen.gz gnttab_max_frames=256

Currently if the issue is encountered, it will be logged to the system log with:
xen/grant-table: max_grant_frames reached ...

To verify the current number of used grants:
# xl debug-keys g; xl dmesg

(XEN) gnttab_usage_print_all [ key 'g' pressed
(XEN)       -------- active --------       -------- shared --------
(XEN) [ref] localdom mfn      pin          localdom gmfn     flags
(XEN) grant-table for remote d0 (v1)
(XEN)   1 frames (64 max), 5 maptrack frames (1024 max)
(XEN) no active grant table entries
(XEN)       -------- active --------       -------- shared --------
(XEN) [ref] localdom mfn      pin          localdom gmfn     flags
(XEN) grant-table for remote d1 (v1)
(XEN)   3 frames (32 max), 0 maptrack frames (1024 max)
(XEN) [0x000]      0 0x65d24 0x00000002          0 0x0fefff 0x19
(XEN) [0x001]      0 0x65d27 0x00000002          0 0x0feffc 0x19
(XEN) [0x008]      0 0x53632 0x00000002          0 0x036c32 0x19
(XEN) [0x300]      0 0x06b74 0x00000001          0 0x03b774 0x19
(XEN) [0x301]      0 0x06fef 0x00000001          0 0x03b3ef 0x19
(XEN) [0x500]      0 0x53254 0x00000001          0 0x037054 0x19
(XEN) [0x501]      0 0x53255 0x00000001          0 0x037055 0x19
(XEN) gnttab_usage_print_all ] done

To change the value for guests add "max_grant_frames=xx" to their configuration file or add the entry to "/etc/xen/xl.conf" in order to set the default for guests without having "max_grant_frames" in their configuration.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

  • Document ID:7018590
  • Creation Date: 06-Feb-2017
  • Modified Date:03-Mar-2020
    • SUSE Linux Enterprise Server

< Back to Support Search

For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback[at]suse.com

tick icon

SUSE Support Forums

Get your questions answered by experienced Sys Ops or interact with other SUSE community experts.

tick icon

Support Resources

Learn how to get the most from the technical support you receive with your SUSE Subscription, Premium Support, Academic Program, or Partner Program.

tick icon

Open an Incident

Open an incident with SUSE Technical Support, manage your subscriptions, download patches, or manage user access.