kswapd is using a lot of CPU cycles

This document (7017605) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 11 Service Pack 1 (SLES 11 SP1)
SUSE Linux Enterprise Server 11 Service Pack 2 (SLES 11 SP2)
SUSE Linux Enterprise Server 11 Service Pack 3 (SLES 11 SP3)
SUSE Linux Enterprise Server 11 Service Pack 4 (SLES 11 SP4)
SUSE Linux Enterprise Server 12 Service Pack 1 (SLES 12 SP1)

Situation

If the system is running into low memory conditions it may happen the kernel thread 'kswapd'
is using a lot of CPU cycles.

Resolution

Either reduce the memory pressure on this system or increase the amount of memory this system
can use.

Cause

Usually the kswapd sleeps. If the amount of free memory runs below a low watermark (pages_low), kswapd is woken up. It then tries to reclaim pages until a high watermark has been reached. If the system is under memory pressure, it can cause the kswapd scanning the available memory over and over again. This has the effect of kswapd using a lot of CPU cycles.

Additional Information

For every memory node the system has configured, one kswapd will be used. The node number is the suffix of kswapd.
E.g.
for node0 the name of the kswapd is kswapd0, for node1 it will be kswapd1 and so on.
To determine what kswapd is doing you can run the following command on the system:

cat /proc/`pidof kswapdN`/stack

Also, every Node is divided into zones. Usually three zones exist per node,
DMA, DMA32 and Normal. Normal memory allocations will be served from zone Normal. The zones can be examined with:

linux-ymxf:/ # cat /proc/zoneinfo
Node 0, zone      DMA
pages free     2497
        min      92
        low      115
        high     138
        scanned 0
        spanned 4095
        present 3998
        managed 3977
    nr_free_pages 2497
    nr_alloc_batch 23
    nr_inactive_anon 1
    nr_active_anon 187
    nr_inactive_file 520
    nr_active_file 575
    nr_unevictable 0
    nr_mlock     0
    nr_anon_pages 186
    nr_mapped    111
    nr_file_pages 965
    nr_dirty     0
    nr_writeback 0
    nr_slab_reclaimable 76
    nr_slab_unreclaimable 34
    nr_page_table_pages 36
    nr_kernel_stack 2
    nr_unstable 0
    nr_bounce    0
    nr_vmscan_write 0
    nr_vmscan_immediate_reclaim 0
    nr_writeback_temp 0
    nr_isolated_anon 0
    nr_isolated_file 0
    nr_shmem     2
    nr_dirtied   226
    nr_written   94
    nr_pages_scanned 0
    numa_hit     28937
    numa_miss    0
    numa_foreign 0
    numa_interleave 0
    numa_local   28937
    numa_other   0
    nr_anon_transparent_hugepages 0
    nr_free_cma 0
        protection: (0, 1877, 1877, 1877)
pagesets
    cpu: 0
              count: 0
              high: 0
              batch: 1
vm stats threshold: 2
all_unreclaimable: 0
start_pfn:         1
inactive_ratio:    1
Node 0, zone    DMA32
pages free     268210
        min      11171
        low      13963
        high     16756
        scanned 0
        spanned 520160
        present 520160
        managed 481700
    nr_free_pages 268210
    nr_alloc_batch 1015
    nr_inactive_anon 2645
    nr_active_anon 65977
    nr_inactive_file 58455
    nr_active_file 57500
    nr_unevictable 20
    nr_mlock     20
    nr_anon_pages 65859
    nr_mapped    16298
    nr_file_pages 104213
    nr_dirty     0
    nr_writeback 0
    nr_slab_reclaimable 9830
    nr_slab_unreclaimable 5482
    nr_page_table_pages 2397
    nr_kernel_stack 187
    nr_unstable 0
    nr_bounce    0
    nr_vmscan_write 0
    nr_vmscan_immediate_reclaim 0
    nr_writeback_temp 0
    nr_isolated_anon 0
    nr_isolated_file 0
    nr_shmem     2783
    nr_dirtied   27274
    nr_written   12749
    nr_pages_scanned 0
    numa_hit     4167496
    numa_miss    0
    numa_foreign 0
    numa_interleave 9706
    numa_local   4167496
    numa_other   0
    nr_anon_transparent_hugepages 77
    nr_free_cma 0
        protection: (0, 0, 0, 0)
pagesets
    cpu: 0
              count: 148
              high: 186
              batch: 31
vm stats threshold: 10
all_unreclaimable: 0
start_pfn:         4096
inactive_ratio:    3

The most interesting lines are:

pages free     2497
        min      92
        low      115
        high     138
[...]
    nr_free_pages 2497

free: is the number of memory pages (a 4KiB) available for allocations.
min : is the lowest watermark. If the number of free pages fall below this threshold, memory allocations will be done synchronously. This means any process doing memory allocations is going to free pages by itself.
low : when this number of free pages is reached, kswapd is woken up to start freeing pages.
high: once kswapd has been woken to start freeing pages it will continue freeing pages until this watermark has been reached. After this watermark has been reached kswapd will go back to sleep.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.