DEV Community

Cover image for Why Servers Crash Under High Traffic: The TCP TIME_WAIT Trap
Binary Dose
Binary Dose

Posted on Originally published at binarydose.in

Why Servers Crash Under High Traffic: The TCP TIME_WAIT Trap

#backend #networking #tutorial #webdev

Why Servers Crash Under High Traffic: The TCP TIME_WAIT Trap

When a backend service or microservice opens thousands of outbound HTTP connections per second, CPU utilization might sit at 10%, yet the server suddenly crashes with:

connect() failed (99: Cannot assign requested address)
Enter fullscreen mode Exit fullscreen mode

The culprit is not a memory leak or a slow database query. It is the operating system's TCP TIME_WAIT state.

⚡ The TCP 4-Way Teardown Flow

When an endpoint initiates an active close:

Client (Active Close)                 Server (Passive Close)
       │                                       │
       │─────── FIN (seq=100) ────────────────>│ (CLOSE_WAIT)
       │<────── ACK (ack=101) ─────────────────│
       │                                       │
       │<────── FIN (seq=300) ─────────────────│ (LAST_ACK)
       │─────── ACK (ack=301) ────────────────>│ (CLOSED)
       │
  [TIME_WAIT] ⏳ (Holds socket for 2 × MSL: ~60–120s)
       │
    (CLOSED)
Enter fullscreen mode Exit fullscreen mode

The host that sends the final ACK does NOT immediately close the socket. It enters TIME_WAIT and holds the (Source IP, Source Port, Dest IP, Dest Port) tuple in the kernel socket table for 2 × MSL (Maximum Segment Lifetime).

🧠 Why Does TIME_WAIT Exist?

It serves two fundamental protocol guarantees:

  1. Clean Final ACK Delivery: If the client's final ACK drops on the wire, the server retransmits its FIN. If the client closed instantly, it would reply with an RST (connection reset error) instead of completing an orderly shutdown.
  2. Ghost Packet Draining: Old TCP segments delayed in slow router buffers could arrive minutes later. If the OS reassigned that identical IP:Port pair immediately, ghost packets from the previous session would corrupt the new data stream!

🛠️ The Ephemeral Port Exhaustion Trap

Linux provides a limited pool of ephemeral outbound ports (typically ~28,000 ports between 32768 and 60999).

If your microservice opens 500 connections/sec without connection reuse:

Sockets in TIME\_WAIT = 500 × 60 seconds = 30,000 sockets!
Enter fullscreen mode Exit fullscreen mode

All local outbound ports are exhausted!

The Production Fix:

  • Enable HTTP Keep-Alive: Reuse persistent connections via connection pooling (HTTP clients like Axios, Go's http.Transport, or Python requests.Session).
  • Tune Linux Socket Reuse: 
  sysctl -w net.ipv4.tcp_tw_reuse=1
Enter fullscreen mode Exit fullscreen mode

🚀 Trace the Full TCP State Machine Interactively

You can test state transitions, inject lost packets, and simulate the 2MSL timer interactively on Binary Dose.

Top comments (0)