I Raised ulimit for 48 Hours. TIME_WAIT Had Already Eaten the Port Range

Have you ever watched a client drown in timeouts while CPU, RAM, and the remote API all looked bored? I spent two days doing exactly that, and I kept treating the wrong ceiling as the bottleneck. File descriptors and ephemeral ports fail in similar English. They are not the same resource, and mixing them up makes every “fix” look reasonable until the next burst.

This is a 48-hour field notebook, not a war story with fake graphs. I will show what I tried, what actually broke, a localhost reproduction you can run, and the checks I would repeat. I will also mark the parts that are a lab procedure rather than a production change.

The question I should have asked first

When a short-lived TCP client starts failing under churn, what ran out? Was it process file descriptors, or the local port range sitting in TIME_WAIT? Those two limits live in different files, they show up in different counters, and they want different repairs. Why did I keep raising ulimit -n like it was a personality trait?

The workload was a Python worker that opened a fresh TCP connection for every tiny request. Under a slow soak it looked fine. Under a burst it started throwing OSError: [Errno 99] Cannot assign requested address on Linux, mixed with timeouts that made the remote service look guilty. Does that error message scream “ports,” or does it just sound like networking weather?

Hours 0–12: the confident wrong loop

I did the usual comfort checks first, because they are cheap and they usually flatter the hypothesis you already like.

• Hit the API with curl from the same box and called it healthy.
• Raised the HTTP pool size, then raised it again, because more sockets feel like capacity.
• Ran ulimit -n, panicked at 1024, and bumped the nofile limit in the shell.
• Restarted the worker, watched one green burst, and declared the patient cured.

Here is the trap. A higher file-descriptor cap lets you hold more sockets. It does not create more four-tuples when the local port range is exhausted. If each request connects, sends, and closes, the client owns TIME_WAIT. After that, new connects fail even while lsof still looks roomy. Have you ever “fixed” a leak by raising a ulimit and then shipped the leak?

A model suggestion I asked for, later in the day, also pointed at ulimit and connection-pool max size. That answer is common, fluent, and incomplete. Vibe-shaped advice is not the same as measuring which table filled up.

Hours 12–30: the free second environment, and the first honest counter

My laptop was a junk drawer of leftover sockets, exported proxies, and a shell profile that has lied to me before. I needed a box that did not inherit that mess. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used MonkeyCode’s free model access to list candidate checks, then ran those checks on the free server option so I was not debugging inside the same polluted session.

I did not ask the model to “optimize networking.” I asked it for commands that distinguish file descriptors from local ports, then I executed them myself. That split matters. The model can draft a checklist. It cannot see your ss output unless you paste it, and it will still happily conflate ENOBUFS, EMFILE, and EADDRNOTAVAIL if you let it.

The commands that finally changed the story were boring:

ulimit -n
ss -s
ss -tan state time-wait | wc -l
ss -tan state established | wc -l
cat /proc/sys/net/ipv4/ip_local_port_range
cat /proc/sys/net/ipv4/tcp_max_tw_buckets
python3 -c "import resource; print(resource.getrlimit(resource.RLIMIT_NOFILE))"

Read those as a set, not as a vibe. ulimit -n is the process ceiling for open files, including sockets. ip_local_port_range is the pool of source ports the kernel will pick. TIME_WAIT entries occupy that pool until 2MSL expires, which on many Linux defaults is on the order of a minute. Do not memorize my numbers. Print yours.

On the clean box, nofile was already large enough to be uninteresting. TIME_WAIT was not. After a burst of connect/close, ss -s showed a TIME_WAIT pile, and new clients failed with Cannot assign requested address while established counts stayed modest. Raising the pool size made the pile grow faster. That is the moment the story flipped. More concurrency was not capacity. It was a shorter path to an empty port range.

Hours 30–48: a reproduction I could re-run

I wanted something smaller than the real worker. The snippet below is a lab procedure, not production code. Run it only on localhost, and stop it if the machine feels unhappy. It is meant to show client-side TIME_WAIT, not to benchmark anything.

Terminal A — tiny server

# lab_echo_server.py
import socket

HOST, PORT = "127.0.0.1", 18080

sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
sock.bind((HOST, PORT))
sock.listen(128)
print(f"listening on {HOST}:{PORT}", flush=True)

while True:
    conn, _addr = sock.accept()
    try:
        conn.recv(16)
        conn.sendall(b"ok")
    finally:
        conn.close()

Terminal B — client that throws connections away

# lab_burn_ports.py
import socket
import sys
import time

HOST, PORT = "127.0.0.1", 18080
count = int(sys.argv[1]) if len(sys.argv) > 1 else 8000
errors = {}

started = time.monotonic()
for i in range(count):
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    s.settimeout(0.5)
    try:
        s.connect((HOST, PORT))
        s.sendall(b"ping")
        s.recv(16)
    except OSError as exc:
        key = getattr(exc, "errno", None) or type(exc).__name__
        errors[key] = errors.get(key, 0) + 1
    finally:
        s.close()

elapsed = time.monotonic() - started
print(f"attempted={count} elapsed_s={elapsed:.2f} errors={errors}")

Run it like this:

python3 lab_echo_server.py
# other terminal
python3 lab_burn_ports.py 20000
ss -tan state time-wait '( sport = :18080 or dport = :18080 )' | wc -l

Watch three outcomes, not one. First, the error dict: 99 on Linux is EADDRNOTAVAIL, which is the port-range story. Second, EMFILE / errno 24, which is the nofile story. Third, timeouts with almost no errors, which often means the peer or the accept queue, not the local range. If you cannot tell those three apart, you are still guessing.

I also kept a one-line “hold the sockets open” variant, because people conflate “too many open files” with “too many recent closes.”

# lab_hold_fds.py — lab only; do not background this and walk away
import socket, sys, time

socks = []
for i in range(int(sys.argv[1])):
    s = socket.socket()
    s.connect(("127.0.0.1", 18080))
    socks.append(s)
print(f"holding {len(socks)} sockets; ^C to drop")
time.sleep(3600)

If lab_hold_fds.py dies with EMFILE while TIME_WAIT stays small, your ulimit hypothesis finally earned its keep. If lab_burn_ports.py dies with EADDRNOTAVAIL while ulimit -n still has headroom, stop touching nofile.

Decision table I wish I had on hour one

Symptom	Tempting story	Check before you “tune”	Repair that actually matches
EMFILE / errno 24	kernel is mean	ulimit -n, /proc/<pid>/limits, `ls /proc//fd \	wc -l`
EADDRNOTAVAIL / errno 99 on connect	API or DNS	ss -s, TIME_WAIT count, ip_local_port_range	reuse connections, HTTP/2 or a pool, stop connect/close per request
Timeouts, peer healthy	remote slowness	established vs TIME_WAIT, server accept queue, client retries	backoff, idempotency, measure RTT; do not widen the pool blindly
Works after a 70s pause	“the cloud recovered”	2MSL / TIME_WAIT expiry	you waited for ports, you did not fix churn
AI says “increase ulimit”	one knob to rule them	ask which errno you have	refuse the knob until the counter agrees

Print the table next to the errno. The table is the artifact. The kernel knobs are not souvenirs.

What I would repeat, and what I will not cargo-cult

I would repeat the split between hold and churn. Holding sockets tests nofile. Churning sockets tests the port range and TIME_WAIT. I would repeat capturing ss -s before changing any config. I would repeat using a second environment when my interactive shell has a history of smuggling variables into the crime scene.

I would not repeat these “fixes” without a written reason:

1. Raising ulimit -n because a blog mentioned it next to “too many connections.”
2. Growing a connection pool to hide connect/close-per-request in application code.
3. Setting tcp_tw_reuse because a checklist from 2014 said so.
4. Looking for tcp_tw_recycle at all. That sysctl was a footgun and is gone from modern Linux. If a model still recommends it, that is a stale training ghost, not a runbook.
5. Treating localhost results as a capacity number you can publish. This lab proves a mechanism. It does not measure your production NIC.

The durable application fix is almost always boring. Keep a pool. Send many requests over one connection. Prefer HTTP/2 multiplexing when the protocol allows it. Close sockets as a last resort, not as a lifestyle. If you must reconnect, cap concurrency so the local range cannot be drained faster than TIME_WAIT expires.

A small pool that is actually reused looks like this, still labeled as an example rather than a library:

# example only: reuse one connection instead of burning ports
import http.client

conn = http.client.HTTPConnection("127.0.0.1", 18080, timeout=2)
try:
    for _ in range(1000):
        conn.request("POST", "/", body=b"ping")
        resp = conn.getresponse()
        resp.read()
finally:
    conn.close()

Compare ss -tan state time-wait after that loop with the burn script. If TIME_WAIT barely moves, you learned the lesson. If it still explodes, you are not reusing what you think you are reusing. Is the client building a new HTTPConnection inside the loop? Is a helper calling close() on every response because a snippet told it to be “clean”?

Limitations, and who should not use this notebook

This write-up assumes you can read Linux ss and /proc/sys. Windows port exhaustion is real, but the counters and the error text are different, so do not copy these commands into a PowerShell window and call it science. macOS will also disagree with Linux about names and defaults. Verify on the kernel you actually run.

Do not use this as permission to poke sysctls on a shared host. ip_local_port_range and TIME_WAIT policy are machine-wide. If you do not own the box, you do not get to widen the range to paper over an application that reconnects like a firehose. Containers can also lie: the view of ss inside a netns is not always the view on the host.

Do not use a coding model as a production network engineer. Free model access is useful for drafting a differential diagnosis: nofile versus ports versus accept queue versus DNS. It is a bad place to copy sysctl values you have not read about for your kernel. I kept the model on checklist duty and kept the counters in my own terminal. That boundary is the whole method.

Who should skip this approach? Anyone whose traffic is already multiplexed and whose TIME_WAIT is idle. Anyone debugging TLS handshakes, HTTP/2 settings, or application deadlocks and hoping a port-range article will save them. Anyone who cannot reproduce on localhost and wants a magic production knob instead of an errno.

Closing the notebook

Two days taught me one cheap question. When the client fails, which table filled: the fd table, or the local port range? If you cannot answer with a counter, you are negotiating with a ghost. Raise nofile only for held sockets. Reuse connections when the pain is TIME_WAIT. Ask any assistant for checks, then make it shut up until ss has spoken.

If you need a second shell that is not wearing your laptop’s habits, the free server option is a convenient place to rerun the lab scripts above. Keep the disclosure in mind, keep the sysctls unread until the errno is known, and keep the 48 hours for a bug that still survives those two counters.