Resolving conntrack table full error messages: 'nf_conntrack: table full, dropping packets'

Issue

When investigating a connectivity issue, you may experience errors like the below in the system logs:

nf_conntrack: table full, dropping packets

This error indicates the connection tracking table size has been exhausted. This can manifest with different symptoms, such as intermittent or consistent network timeouts.

The conntrack table keeps state on open connections that the kernel is translating. This occurs often in a Kubernetes cluster when pods access an external endpoint, or another service within the cluster. These scenarios use NAT and stateful firewall rules which are maintained as entries in the conntrack table.

Investigation

By default, the table size is calculated based on the memory allocated to the node. This does not fit all workloads demands, for example in a microservice environment typically a higher number of inter-service connections could be expected without consuming a high amount of memory.

To output the current max table size:

cat /proc/sys/net/netfilter/nf_conntrack_max

To get a point in time count of the current entries in the table:

cat /proc/sys/net/netfilter/nf_conntrack_count

Note: With the conntrack package installed, you can also use conntrack -C

If the nf_conntrack_count and nf_conntrack_max are close, it is indicating that the current workload requires a larger table size.

If the current number of entries are not approaching the table size, this could indicate that a burst of workload was experienced historically, in a containerized environment this can be common. For example, if the high-traffic Pods may now running on different nodes.

Resolution

Increasing the conntrack table size is achieved with sysctl.

Calculate a higher value, this can be applied to the node immediately with:

sysctl -w net.netfilter.nf_conntrack_max=<value>

To persist through reboot, add the tunable to either /etc/sysctl.conf, or a specific config file in /etc/sysctl.d.

For example, if your Linux distribution follows the /etc/sysctl.d/ directory structure:

echo "net.netfilter.nf_conntrack_max=<value>" > /etc/sysctl.d/10-conntrack-max.conf
sysctl -p /etc/sysctl.d/10-conntrack-max.conf

This creates a new config file to set the table size at each boot.

Additionally, if you configure nodes with configuration management, UserData, or build custom images etc., you may wish to add this to your usual approach to configure this for future nodes.