I Blamed Sticky Sessions for 48 Hours. The Pool Still Held the Old Socket.

I spent forty-eight hours convinced our load balancer was sticky in the worst possible way after a quiet rolling deploy. After every rolling restart, a slice of traffic still parsed JSON fields I had already deleted from the handlers. Does that sound like a platform bug, or like a client process that never bothered to hang up?

These are field notes from that debugging loop, not a dashboard dump with invented graphs or conversion rates. I will show what I tried first, what broke in the harness, and the checks I would run again tomorrow. If you strip every product mention out of this post, the socket still lies, and the repro still runs.

Hour 0–8: I blamed the wrong layer

The symptom looked like classic stickiness because only some callers saw the new response header after cutover. A canary instance returned the new shape, while long-running jobs kept parsing the payload I had just removed. I stared at target health until the new boxes were green and the old boxes were marked unused. Was the balancer ignoring connection drain, or was I watching the wrong layer on purpose?

What I tried before I questioned the client:

1. I grepped every Terraform file for stickiness, cookie affinity, and source-IP hashing, then found none enabled.
2. I watched target health flip from draining to unused, which should have meant no live sockets on the old process.
3. I curled the public hostname from my laptop and got the new header every time, which made the jobs look cursed.
4. I blamed eventual consistency out loud, which is usually how I mark a session that has already gone sideways.

The laptop curls were the trap hiding in plain sight, and I should have seen that earlier. A short curl opens one connection, reads the body, and exits like a polite guest who never sits down. My workers were not polite guests, and they had been sitting on the same chairs since process boot.

Hour 8–24: The worker was a Session

The jobs used a shared requests.Session created at import time, then handed around like a global database handle. That object is a connection pool with opinions, not a fancy urllib wrapper you can ignore when deploys get weird. Once a TCP connection is idle but still alive, the next request reuses it and never asks the resolver again. Ask yourself this: if the pool already holds a live socket, why would it open a brand new one?

It would not, and for hours I kept testing with tools that always opened a brand new one. Here is the shape of the worker I was actually running, simplified and labeled as a repro, not production code.

# repro_session_pool.py
# Labeled example: a long-lived Session against a backend you can restart.
import os
import time
import threading
import requests

BASE = os.environ.get("API_BASE", "http://127.0.0.1:8080")
PATH = os.environ.get("API_PATH", "/whoami")

session = requests.Session()  # created once, like a worker module import


def loop(name: str, delay: float) -> None:
    while True:
        try:
            resp = session.get(f"{BASE}{PATH}", timeout=(0.5, 2.0))
            print(f"{name} status={resp.status_code} body={resp.text!r}")
        except Exception as exc:
            print(f"{name} error={type(exc).__name__}: {exc!r}")
        time.sleep(delay)


if __name__ == "__main__":
    threading.Thread(target=loop, args=("worker", 1.0), daemon=True).start()
    print("pid", os.getpid(), "GET", f"{BASE}{PATH}")
    time.sleep(10_000)

A one-shot script never keeps the pool warm long enough to lie to you, which is why laptops look innocent. If your repro exits in two seconds, you are not reproducing a worker, and you are not reproducing this bug.

The backend that tells you which generation you hit

I needed a server that printed a generation string, then I needed to replace that generation without changing the hostname. A tiny http.server subclass is enough for the local cheat, and you should run it where your workers actually run. If the workers live on Linux, do not trust a laptop TCP stack to keep idle sockets in the same mood.

# whoami_server.py
# Labeled example: restart this process to simulate a cutover.
import os
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

GEN = os.environ.get("GEN", "old")
PORT = int(os.environ.get("PORT", "8080"))


class Handler(BaseHTTPRequestHandler):
    protocol_version = "HTTP/1.1"  # keep-alive on

    def do_GET(self) -> None:
        body = f"gen={GEN} pid={os.getpid()}\n".encode()
        self.send_response(200)
        self.send_header("Content-Type", "text/plain")
        self.send_header("Content-Length", str(len(body)))
        self.send_header("Connection", "keep-alive")
        self.end_headers()
        self.wfile.write(body)

    def log_message(self, fmt: str, *args) -> None:
        print("srv", GEN, args[0])


if __name__ == "__main__":
    ThreadingHTTPServer(("0.0.0.0", PORT), Handler).serve_forever()

Commands I keep in the notes, because I will forget the order the next time this happens:

# terminal A: generation "old"
GEN=old PORT=8080 python whoami_server.py

# terminal B: the clingy client
API_BASE=http://127.0.0.1:8080 python repro_session_pool.py

# terminal C: watch the socket stay ESTABLISHED
ss -tpn | grep 8080

# later: stop A, start generation "new" on the same port
GEN=new PORT=8080 python whoami_server.py

# after the bounce, ask two different questions
ss -tpn dst :8080
getent ahosts 127.0.0.1

If the client still prints gen=old after the restart, you do not have a balancer problem sitting in Terraform. You have a corpse socket, or you have a reconnect that rebound the same port and looked like loyalty. Wait, rebound the same port as a local cheat, which is useful, but it is not the same as a DNS cutover. Real cutovers change the IP behind a name, and keep-alive is even more dishonest there, because the TCP session never moves.

getent answers what a brand new socket would do with a name. ss answers what you are still holding. Those are different questions, and I asked only the first one for far too long.

Hour 24–36: What broke when I tried to reproduce it

Local macOS lied in three separate ways, and I want you to expect that instead of assuming your laptop is production. My laptop closed idle sockets faster than the worker boxes, so the pool looked healthy whenever I watched it. My shell also recycled TIME_WAIT in a pattern that did not match the container image, which scrambled the ss output. Have you ever spent twenty minutes comparing netstat flags to ss flags, and then realized you were arguing with yourself?

What broke, in order, when the harness was too polite to be guilty:

• The first harness used http.client without a Session, so every request was a new handshake and the bug vanished.
• The second harness reused a Session but hit Connection: close from a framework default, so the pool never pooled.
• The third harness ran for thirty seconds, which is not long enough for idle keepalive to become the main character.
• Killing the server with Ctrl+C sometimes sent a RST, which woke the client and forced a reconnect that hid the stale-IP case.

See the pattern in that list, because I kept writing a demo that was too polite to stay guilty overnight. I needed a long-lived Linux process, and that is where MonkeyCode actually participated instead of sitting in a slogan.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I needed a cheap box that could sit overnight with the client loop running, plus a second process I could kill and replace. MonkeyCode's free server option was enough for that overnight repro, and free model access helped me draft the harness. Docs say pooling exists, but they do not show you the ss line that quietly survives a rolling deploy.

A decision table I wish I had on hour two

Symptom after cutover	Short curl from a laptop	Long-lived Session / pool	What to inspect first
All clients see new behavior	New body	New body	You probably do not have this bug
Only workers see old behavior	New body	Old body or old header	Keep-alive sockets, ss -tpn, process age
Everyone sees old behavior	Old body	Old body	Wrong build, cache, or a name that never moved
Intermittent mix	Mix	Mix	Multiple pool entries, retries, more than one replica
Errors then recovery	Reset or 502, then new	Brief errors, then old or new	RST versus FIN, drain timeout versus pool idle timeout

If your row is laptop fine, worker stale, stop talking to the balancer team for one hour. You can apologize later if the table is wrong, and that apology is cheaper than another night of dashboards.

Hour 36–48: FIN, RST, and the timeout nobody documented together

The last night was about why some restarts cured the client immediately while other restarts left it talking to ghosts. When the old process closes with a FIN, the client should see the next request fail and then reconnect. When a firewall or balancer silently drops idle packets, the client can keep writing into a black hole until timeout. Which one did I have? I had both, because drain settings and process kills are not the same failure, even when dashboards use the same red color.

I started logging elapsed time and exception class on every worker request, which is boring and also how the night finally ended.

# labeled example: split connect versus read, print the failure class
import time
import requests


def timed_get(session: requests.Session, url: str) -> None:
    t0 = time.monotonic()
    try:
        resp = session.get(url, timeout=(0.5, 2.0))
        dt = time.monotonic() - t0
        print(f"ok {resp.status_code} {dt:.3f}s {resp.text!r}")
    except requests.RequestException as exc:
        dt = time.monotonic() - t0
        print(f"err {type(exc).__name__} {dt:.3f}s {exc}")

A single integer timeout hides whether you are stuck opening a new socket or stuck reading from a socket that is already doomed. Connect timeout versus read timeout matters here, and I had been using one number like a person who enjoys confusion. After that change, the log line told me when I was holding a corpse and when I was racing a bind.

The fix I would repeat, with the parts that still bite

I do not rip Sessions out of every codebase, because connection reuse is good until a name's backends become a rotating cast. The fix is to make reconnect policy explicit, then prove it with ss, not with hope. Pick a lever on purpose, and write down which deploy event is allowed to keep a socket.

# labeled example: three levers, choose one per call path
import requests
from requests.adapters import HTTPAdapter


def make_session(*, recycle: bool) -> requests.Session:
    sess = requests.Session()
    adapter = HTTPAdapter(pool_connections=4, pool_maxsize=8, max_retries=0)
    sess.mount("http://", adapter)
    sess.mount("https://", adapter)
    if recycle:
        # Force Connection: close so the pool cannot keep a pre-deploy socket.
        sess.headers["Connection"] = "close"
    return sess

Practical rules I would repeat:

1. Create the Session at boot for performance, but rebuild it after a known deploy event if the process is a long worker.
2. Set idle timeouts on the pool and the proxy separately, and do not assume those two clocks are friends.
3. On cutover-sensitive calls, skip reuse: a one-off requests.get or Connection: close is boring and correct.
4. Confirm with ss -tpn or lsof -iTCP -sTCP:ESTABLISHED before and after the restart, because logs will lie first.

# before kill
ss -tpn dst :8080

# after the new process bind
ss -tpn dst :8080

# if you use a hostname instead of a literal IP
getent ahosts api.internal.example

Repro test plan you can run without production traffic

This is the sequence I would run again before paging another team. It is a test plan, not a claim that I measured a fleet.

1. Start whoami_server.py with GEN=old and leave it running until ss shows ESTABLISHED from the client.
2. Start repro_session_pool.py and wait long enough that several loops reuse the same connection instead of reconnecting.
3. Confirm the client prints gen=old and that ss -tpn dst :8080 still lists the worker pid.
4. Stop the server in two ways, once with a clean SIGTERM and once with a hard kill, because FIN and RST teach different lessons.
5. Start GEN=new on the same port, then watch whether the client body changes on the next tick or several ticks later.
6. Repeat the whole loop with Connection: close on the Session, and expect the body to follow the new generation after the first error or the first new handshake.

If step five stays on gen=old while step six moves, you found the pool. If both stay old, you are not on the host you think you are.

Limitations, and who should not copy this blindly

This harness is a teaching artifact, not a failover specification for a multi-region client. Two processes bouncing on 127.0.0.1:8080 do not simulate global DNS, anycast, or a cloud load balancer draining with Connection: close. HTTP/2 and HTTP/3 multiplex in ways this HTTP/1.1 toy will never show you, and I did not test TLS tickets, resumption, or 0-RTT.

Do not use this approach if you already have a service mesh with proven idle timeouts and outlier ejection you can actually read. Do not paste Connection: close onto every internal call in a hot loop without measuring, because you will trade a stale-socket bug for a handshake storm. And do not treat overnight ss output on a spare box as proof of how your production balancer drains.

If your real failure mode is a resolver that never returns, this is the wrong article. I am talking about the socket you already have, not the lookup you have not started.

What I would repeat in the next 48 hours

I would start with a long-lived client, not a laptop curl, and I would print elapsed time next to every response body. I would treat the hostname is the same as a smell, not as comfort, because sameness is how keep-alive hides. I would keep the decision table above the dashboard, because dashboards do not show file descriptors.

Would I still peek at stickiness settings? Yes, for five minutes, with a timer on the desk. After that I would run the two Python files, watch ss, and only then page another team. The balancer can still be guilty, but it should have to earn that accusation.