DEV Community

jun liu
jun liu

Posted on

ECONNRESET at 2 AM: The 5-Minute Drill That Tells You Who Closed the Socket First

Your app logs say ECONNRESET. Someone closed the socket — but who? Your app, the peer, or a middlebox nobody told you about? Most engineers jump straight to tcpdump, capture 40,000 packets, and drown. The faster path is to read the socket states you already have.

Here's the 5-minute drill I run, in order.

Minute 1: Get the state distribution 

ss -tn | awk '{print $1}' | sort | uniq -c | sort -rn
Enter fullscreen mode Exit fullscreen mode

Four states actually tell you something:

State What it means Threshold that matters
TIME-WAIT You closed first (normal churn) Low thousands on a proxy is fine; hundreds of thousands is not
CLOSE-WAIT Peer closed, your app never did > 100 is a bug in your code, not the network
SYN-SENT Your outbound connect got no answer > 100 means a path/firewall problem
SYN-RECV Half-open inbound connections Growing = accept queue full or a scan

The one line I want burned into memory: CLOSE-WAIT piling up is always a local application bug. The peer sent FIN, your process never called close(), and that socket will sit there forever. Find the owner:

ss -tnp state close-wait
Enter fullscreen mode Exit fullscreen mode

Minute 2: Check the kernel counters 

netstat -s | grep -iE 'retrans|reset|listen|overflow'
Enter fullscreen mode Exit fullscreen mode

Three counters matter:

  • TCPRetransFail climbing → packets are being dropped somewhere on the path
  • ListenOverflows growing → your accept queue is full; real SYNs are being refused
  • resets growing steadily → someone is killing connections mid-flight

If all of them are flat, TCP is not your problem — stop reading socket tables and go look at the application.

Minute 3: Attribute the RST

This is the step most people skip, and it's the one that ends the argument about "whose fault" it is:

tcpdump -nn -i any 'tcp[tcpflags] & tcp-rst != 0'
Enter fullscreen mode Exit fullscreen mode

Read the source IP of each RST:

  • RST from your own IP → your application closed the socket while the peer was still sending. Look for a crash or an over-eager timeout in your own logs.
  • Source is the peer → they closed first; you sent into a dead socket.
  • RST from a third IP (or a load balancer) → a middlebox with a short idle timeout is pruning your session. The classic case: your app sends on an idle connection every 90 seconds, but the load balancer's idle timeout is 60 seconds. The LB closed silently; your next packet gets an RST.

That third case is the one I see most, and it's a config mismatch, not a bug: align the LB idle timeout with your app's send interval, or send socket-level keepalives (TCP_KEEPIDLE=60, TCP_KEEPINTVL=10, TCP_KEEPCNT=3 — probes start at 60s and give up after ~90s total).

One real example

A reverse proxy showed ~4,000 CLOSE-WAIT sockets at 2 AM. ss -tnp state close-wait showed every one owned by the same worker process. The app had an error path that returned without closing upstream connections — the peer (the upstream) closed, and the leaked sockets sat there forever. The fix was three lines of try/finally, not a single sysctl. Kernel tuning would have just hidden the leak.

The order that works

  1. ss -s — size the problem.
  2. State distribution (ss -tn | awk ... | uniq -c) → tells you which side is misbehaving.
  3. netstat -s counters → tells you whether the kernel even agrees there's a problem.
  4. Only then tcpdump, and only on the specific host/port pair you care about.

Nginx logs tell you the symptom, ss tells you the state, tcpdump tells you the packet-level truth — in that order. Tuning sysctl before you know which side is doing the unusual thing just moves the symptom.

Tags: tcp, networking, linux, debugging, devops

More field-tested network and server notes: jiunotes · ops notes

Top comments (0)