Mustafa ERBAY

Posted on Sep 14 Originally published at mustafaerbay.com.tr

The Two-Hour Keepalive: 8 of 246 Sockets Read It

#tcp #linux #network #nginx

net.ipv4.tcp_keepalive_time = 7200. The most familiar line in any sysctl file, and the most familiar advice about it: two hours is too long, set it to 600. Before changing it I got curious about something else. How many connections on my server actually read this line?

$ sysctl net.ipv4.tcp_keepalive_time net.ipv4.tcp_keepalive_intvl net.ipv4.tcp_keepalive_probes
net.ipv4.tcp_keepalive_time = 7200
net.ipv4.tcp_keepalive_intvl = 75
net.ipv4.tcp_keepalive_probes = 9

$ ss -Htno state established | wc -l
246
$ ss -Htno state established | grep -c "timer:(keepalive"
96

246 established connections, 96 with a keepalive timer. For the remaining 150, that sysctl line means nothing; whether I write 7200 or 60, not a single probe will ever go out on those sockets. Because tcp_keepalive_time is not a switch, it is a default. The switch belongs to each socket, it is called SO_KEEPALIVE, and its factory setting is off.

This post is about who those 96 are, why the 150 stay in the dark, and why two hours no longer saves anyone. The measurements come from my own server; the experiment from two containers on my Mac.

Being off is the specification itself

In 1989, RFC 1122 described TCP keep-alives as a practice "not universally accepted" and laid down two rules: the application must be able to turn them on or off per connection, and they must default to off. The interval "MUST default to no less than two hours." RFC 9293, which restated TCP in 2022, repeats the same sentences as MUST-24, MUST-25, MUST-27 and MUST-28, and adds MUST-26: keep-alives are not sent while sent data is outstanding. Keep that last one in mind; it comes back in the experiment.

The kernel does exactly this. The timer function in net/ipv4/tcp_timer.c checks the socket's SOCK_KEEPOPEN flag as one of its first steps and leaves if it is not set. If it is set, keepalive_time_when() in include/net/tcp.h computes the period: the socket's own keepalive_time if it has one, otherwise the namespace's sysctl. A socket configured with setsockopt(TCP_KEEPIDLE) never even sees the sysctl.

In short there are three tiers, and sysctl is the bottom one:

If SO_KEEPALIVE is off, nothing happens.
If it is on and the socket has set TCP_KEEPIDLE/TCP_KEEPINTVL/TCP_KEEPCNT, those apply.
If it has not, tcp_keepalive_time/intvl/probes kick in.

Most tuning posts only describe the third tier. The first two are the application's decision, and applications decide very differently.

Whoever owns the timer, the countdown reveals their choice

Grouping ss -tnop output by process revealed four different worlds on the same server. The remaining time on each timer also gives away what each stack asked for, because the countdown starts from at most the idle period:

Process	Sockets	Keepalive	Longest counter seen	Whose decision
nginx	107	none	—	no `so_keepalive` given
kopru-gateway (Go)	53	yes	14 s	Go's default: 15 s
haproxy	38	none	—	no `option clitcpka`
etcd (Go)	14	13 yes / 1 none	25 s	Go, with its own setting
cloudflared (Go)	13	yes	28 s	Go, with its own setting
patroni	6	yes	38 min (2 sockets to PostgreSQL) / 8 s (4 sockets to etcd)	libpq system default / etcd client's own setting
postgres	3	yes	38 min	server `tcp_keepalives_idle = 0` → system
sshd	3	yes	119 min	`TCPKeepAlive yes` → system default
other	9	5 yes / 4 none	—	—

246 sockets in total, 96 with keepalive. Two things stood out.

First, the only sockets actually using the two-hour sysctl are sshd, postgres, and patroni's two connections to PostgreSQL. sshd's three sockets have 118, 91 and 119 minutes left; postgres's three have 38 minutes. These processes turned SO_KEEPALIVE on but left the period to the system; they are the ones that would change if I set tcp_keepalive_time to 600. PostgreSQL's documentation says as much: for tcp_keepalives_idle, "a value of 0 (the default) selects the operating system's default." On the libpq side, keepalives defaults to 1 and the period is again the system's.

Second, everything written in Go took care of itself. kopru-gateway is my code and there is not a single line about TCP keepalive in it; the listener is a plain net.Listen("tcp", addr). Yet every one of its sockets has keepalive, and the longest counter is 14 seconds. Go's net package enables keepalive on every accepted connection when ListenConfig.KeepAlive is zero; the constants in dial.go are 15 s idle, 15 s interval, 9 probes. KeepAliveConfig, added in Go 1.23, lets you change these per socket, but I never touched it and had not noticed until today. The yamux layer running in the same process has its own 30-second keepalive, which is a separate mechanism: that one is an application-level ping, this one is the kernel's probe. Neither knows about the other.

Then there is the side left in the dark: nginx's 107 sockets and haproxy's 38. nginx says that when so_keepalive is omitted from listen, "the operating system's settings will be in effect," which means the default of off. HAProxy behaves the same unless option clitcpka and option srvtcpka (or option tcpka, which enables both) are written down. If any of those 145 sockets silently breaks one day, the kernel will never find out.

There is one more counter for the whole server. nstat -az TcpExtTCPKeepAlive counts probes sent: 383,488 over 3 days and 10 hours of uptime, roughly 1.3 probes per second. My server asks someone "are you there?" every second, and almost all the askers are Go processes.

What ss calls "8.440ms" is 8 seconds

While building the table I was puzzled for a while. ss on Ubuntu 24.04 printed timer:(keepalive,8.440ms,0) for some sockets, and a second later the same socket said 7.191ms. A millisecond counter does not drop 1,249 units in one second.

The answer is in iproute2's source. print_ms_timer() writes the minute, second and millisecond parts separately; for values with no minutes and at most 9 seconds it appended a dot to the seconds part and printed a plain "ms" after the millisecond part. 8.440ms is actually 8 seconds and 440 milliseconds. The bug was fixed in July 2024 by commit 3e807112fd; Ubuntu 24.04's iproute2 6.1.0 predates the fix. The 7.0.0 in the Alpine image I used for the experiment printed the same value as 3.063sec.

A small thing, but enough to lose half an hour wondering whether keepalive fires every 8 milliseconds.

The silent-break experiment

To see when keepalive helps and when it does not, I set up two Python containers on my Mac. The server accepts connections and waits; the client container opens five connections at once. About two seconds after the connections are established, iptables -I INPUT -s <client> -j DROP runs inside the server container: every packet from the client goes silently to the bin, no RST, no ICMP. A pulled cable or a NAT box flushing its table looks exactly like this; some cloud load balancers at least send an RST, and I will get to those.

The five clients:

idle-noka: waits, keepalive off.
idle-ka: waits, SO_KEEPALIVE on, TCP_KEEPIDLE=5, TCP_KEEPINTVL=2, TCP_KEEPCNT=3.
send-noka: sends 1000 bytes per second from the third second on, keepalive off.
send-ka: same sending, keepalive on with the same settings as idle-ka.
send-ut: same sending, no keepalive, but TCP_USER_TIMEOUT=10000.

The result:

[idle-ka]   failed after  11.3s: TimeoutError(110, 'Operation timed out')
[send-ut]   failed after  14.1s: TimeoutError(110, 'Operation timed out')
[idle-noka] failed after  75.0s: TimeoutError('timed out')      # the application's 75 s settimeout
[send-noka] failed after 108.1s: TimeoutError('timed out')      # buffer full at ~33 s + 75 s settimeout
[send-ka]   failed after 108.2s: TimeoutError('timed out')      # same

idle-ka went exactly by the arithmetic: 5 seconds idle, 3 probes 2 seconds apart, ETIMEDOUT at second 11. idle-noka heard nothing from TCP at all; the only reason it fell over at 75 seconds is Python's settimeout. Without that line it would sit in recv forever.

The real lesson is the send-ka line. I had enabled keepalive with the same aggressive settings, and it did nothing; it died at the same moment as send-noka. The 108 seconds have nothing to do with TCP: the send buffer filled up around second 33, send() blocked, and 75 seconds later Python's settimeout went off. The ss -tnoi snapshot at second 60 shows why:

0  30000  172.23.0.3:44442  172.23.0.2:9000  timer:(on,47sec,8)
    ... rto:51456 backoff:8 bytes_sent:10000 bytes_retrans:9000 bytes_acked:1 unacked:1 retrans:1/9

The timer is not keepalive but on: the retransmission timer. bytes_acked:1 and retrans:1/9 also show that the cut came before the first data byte; a single segment was retransmitted nine times and never acknowledged. Because there is unacknowledged data in the queue, the kernel never enters the keepalive branch; the comment on that line in tcp_timer.c is even a little smug: "It is alive without keepalive 8)". In this state the connection's fate is decided by tcp_retries2, default 15 attempts, which the documentation calls "a hypothetical timeout of 924.6 seconds." At the eighth backoff the RTO had already grown to 51 seconds; it would have seen 15 minutes.

send-ut is the antidote. TCP_USER_TIMEOUT says, in milliseconds, how long unacknowledged data may wait; I gave it 10 seconds and it failed at second 14 (the first data went out at second 3 and was never acknowledged). PostgreSQL exposes this as tcp_user_timeout on both the server and the libpq side. Keepalive answers "did the peer die while idle"; user timeout answers "I am sending, is anyone receiving." They are not the same knob, but they are not independent either: when both are set on the same socket, tcp_keepalive_timer decides death by the user timeout duration rather than the probe count; TCP_KEEPCNT is effectively disabled, and a huge user timeout also delays keepalive from reporting the break.

The server side must not be forgotten either. After the client container was deleted, I looked at the server:

$ ss -Htno state established | grep -c 9000
10

All ten connections from both runs are still ESTABLISHED. There is no keepalive on the server side, the peer has vanished, and the kernel will never learn. In real life this is one of the most common explanations for the "connection count only goes up overnight" graph.

Two hours is long for whom

The discussion section of RFC 1122 asks: "if no one is using the connection, who cares if it is still good?" In 1989 that was a reasonable question. In 2026 the boxes in the middle answer it, and none of them has two hours of patience:

AWS Network Load Balancer applies a 350-second idle timeout to TCP flows (configurable between 60 and 6000). After it expires, whichever side sends data receives an RST. The documentation says explicitly that "clients or targets can use TCP keepalive packets to restart the idle timeout"; on TLS listeners the 350 seconds is fixed, and once the LB sees a keepalive it generates its own every 20 seconds toward both legs.
Azure Load Balancer defaults to 4 minutes, with a range of 4 to 100 minutes; whether it sends an RST on expiry depends on the rule's "TCP reset" option, and if that is not on, the connection is quietly forgotten.
Linux conntrack's nf_conntrack_tcp_timeout_established is 432,000 seconds, five days. That box is patient, but when its table fills these entries are the first to go; I covered that in the conntrack capacity post.

A keepalive that waits two hours keeps nothing alive behind an NLB that forgets after 350 seconds. By the time the first probe goes out, the box has long deleted the connection; the probe either falls into silence or eats an RST. That is why I set the idle period to "half of the least patient box in the path": around 150 seconds with an NLB, 100 seconds with Azure. Redis has defaulted to 300 seconds since 3.2.1; its config file gives two reasons, "detect dead peers" and "force network equipment in the middle to consider the connection to be alive." That has always been the whole point.

One more warning, from HAProxy's documentation: keepalive packets never reach the application layer, only the network stack sees them, so keepalives on one side of a proxy are not forwarded to the other side. The client–proxy and proxy–server legs must be protected separately. The end-to-end question "is the other side really responding" belongs to the application, not TCP; the gRPC/HTTP-2 keepalive post covers that layer. SSH's distinction between TCPKeepAlive and ClientAliveInterval is the same duality: the first is a kernel probe, the second a real message over the encrypted channel. sshd -T on my server shows both: tcpkeepalive yes, clientaliveinterval 300.

Every stack has its own knob

Before touching sysctl, you need to know the application's own door. What I verified against the documentation today:

Stack	Enabling	Timing
nginx (inbound)	`listen 443 ssl so_keepalive=on;`	`so_keepalive=30m::10` → 30 min idle, system interval, 10 probes (1.1.11+)
nginx (upstream)	`proxy_socket_keepalive on;` (1.15.6+, default off)	system values
HAProxy	`option clitcpka` / `option srvtcpka`	`clitcpka-idle`, `clitcpka-intvl`, `clitcpka-cnt` (srv equivalents exist)
PostgreSQL server	always on	`tcp_keepalives_idle`, `_interval`, `_count`, `tcp_user_timeout` (0 = system)
libpq / psycopg	`keepalives=1` by default	e.g. `keepalives_idle=60` in the connection string
Redis / Valkey	`tcp-keepalive 300` (default)	single value; 0 disables
Go `net`	on by default	`KeepAliveConfig{Idle, Interval, Count}` (1.23+), default 15 s / 15 s / 9
Node.js	`socket.setKeepAlive(true, ms)`	interval 1 s and count 10 since v12.17; `interval` and `count` arguments in v24.19 / v26.4
systemd socket unit	`KeepAlive=yes`	`KeepAliveTimeSec=`, `KeepAliveIntervalSec=`, `KeepAliveProbes=`

The Node row deserves extra attention. http.globalAgent has shipped with keepAlive: true since v19, and when a socket returns to the pool, the keepSocketAlive function in _http_agent.js calls socket.setKeepAlive(true, 1000). So the outbound HTTP connections of a modern Node service are protected with 1 s idle, 1 s interval, 10 probes. Those 11 seconds matter little to a socket sitting idle in the pool, because globalAgent also gives the same socket a 5-second timeout and destroys an idle socket itself; where the kernel probe really earns its keep is the socket that has sent a request and is waiting for the reply. I once met someone complaining that "my Node app generates too many keepalive packets" without knowing this; it was generating exactly as many as it should.

The container has its own two hours

keepalive_time_when() reads the sysctl through sock_net(sk): the value belongs to the network namespace. And every new namespace starts from constants in tcp_sk_init() in tcp_ipv4.c: TCP_KEEPALIVE_TIME is two hours, 9 probes, 75 seconds. What the host says does not matter. I checked on my server:

$ docker exec vpsman-vps3-edge-1 cat /proc/sys/net/ipv4/tcp_keepalive_time
7200

Had I set 600 on the host, this container would still see 7200. Docker accepts net.* sysctls per container with --sysctl net.ipv4.tcp_keepalive_time=600 (except with --network=host). Do not look for a separate IPv6 knob; net.ipv4.tcp_keepalive_* applies to both families. On the Kubernetes side, net.ipv4.tcp_keepalive_time, _intvl and _probes have been on the "safe" sysctl list since 1.29, with a kernel 4.5+ requirement; they go into the pod's securityContext.sysctls, no kubelet flag needed; for hostNetwork: true pods the net.* sysctls are skipped and the pod sees the node value. Editing sysctl on the node and expecting pod behaviour to change may be the most common mistake in this whole post.

On your own server, in order

Seven questions are enough:

ss -Htnop state established | grep -c "timer:(keepalive" — how does this compare to the total? The difference is the sockets sysctl cannot reach.
Which processes have no timer? nginx and haproxy are the usual suspects; their doors are in the table above.
Who is the least patient box in the path? NLB 350 s, Azure 4 min, corporate firewalls often shorter. Make the idle period half of that.
Is there a flow that sends data and waits for a long time (replication, queues, tunnels)? That needs TCP_USER_TIMEOUT, not keepalive; otherwise 15 minutes of retransmission.
Containers and pods do not see the host sysctl; set it separately with --sysctl or securityContext.sysctls.
Watch the nstat -az TcpExtTCPKeepAlive counter. If it is zero, nobody speaks keepalive; if it suddenly jumps, a library default has changed.
If ss on Ubuntu 24.04 says "8.440ms", read 8 seconds.

Who owns the sysctl

What came out of this measurement was not really the behaviour of a sysctl but the misreading of a sentence. "The keepalive period is two hours" sounds like a property of the system; all the system actually offers is a number to be used if anyone asks. 150 of 246 connections never asked. Of the 96 that did, 88 brought their own period; only 8 sockets actually read the sysctl, three of them sshd's.

That is how factory settings work: each layer carries its own default, and when defaults stack up you get behaviour nobody decided on. Go says 15 seconds, Node says 1 second, the kernel says 2 hours, nginx says nothing. All on the same server, at the same time, under the same sysctl. Measuring who asked before explaining a behaviour with "that's just how the system is" usually teaches more than changing the setting does.

DEV Community