ip_conntrack: table full, dropping packet

Is your Linux system complaining about this error? This is because you have iptables enabled, and have rules that are set to act based on packet state and your server is trying to handle more connections than it’s configured to handle at once. Here is how to see what number it’s currently tracking

# cat /proc/sys/net/ipv4/netfilter/ip_conntrack_count

If you want a brainless fix to buy you some time so you can finish this article, try this:

# echo 131072 > /proc/sys/net/ipv4/ip_conntrack_max

That should give you some short term relief in exchange for some memory used by the kernel.

However, it’s probably a good thing to know what ip_conntrack is in the first place, and why it fills up. If you run an iptables firewall, and have rules that act upon the state of a packet, then the kernel uses ip_conntrack to keep track of what state what connections are in so that the firewall rule logic can be applied against them. If you have a system that’s getting a lot of network activity (high rates of connections, lots of concurrent connections, etc) then the table will accumulate entries.

The entries remain until an RST packet is sent from the original IP address. If you have a flaky network somewhere between you, and the clients accessing your server, it can cause the RST packets to be dropped due to the packet loss, and leave orphaned entries in your ip_conntrack table. This can also happen if you have a malfunctioning switch or NIC card… not necessarily a routing problem out on the internet somewhere.

Typically when I’ve seen this trouble crop up is when a server is the target of a DDoS attack. Filling up the ip_conntrack table is a relatively easy way to knock a server off line, and attackers know this.

You can get short term relief by increasing the size of the table. However, these entries are held in memory by the kernel. The bigger you make the table, the more memory it will consume. That memory could be used by your server to serve requests if you really don’t need the stateful firewall capability. Don’t waste resources on this feature if you really don’t need it.

Another option to consider is turning OFF iptables rules that use ip_conntrack so the state table is not used at all. Anything with “-m state” or “-t nat” can be turned off. If you want to just flush all your iptables rules you can do an “iptables -P” to set a default allow policy and “iptables -F” to flush all the rules. On an RHEL or CentOS system you can just do “service iptables stop”.

Once iptables is no longer using ip_conntrack, you can reclaim the memory the table was using by unloading the related kernel modules.

modprobe -r ipt_MASQUERADE
modprobe -r iptable_nat
modprobe -r ipt_state
modprobe -r ip_conntrack

Then you will have an empty ip_conntrack that will stay empty. I mention this because a lot of sysadmins have hordes of iptables rules installed as a matter of course, and don’t recognize the downside of having them present. You can still use iptables, but you can avoid the use of ip_conntrack by simply not using rules based on stateful logic.