I Profiled a Sleeping Worker for 48 Hours. The Client Never Set timeout.

Have you ever stared at a still-running worker that never finishes even a single job? I did that for two long days and blamed everything except the HTTP client itself. The queue looked healthy, the process stayed alive, and CPU sat near zero like the box had fallen asleep. What I finally reproduced was boring, well documented, and still emitted by coding assistants that write a simple GET helper.

These notes are a lab notebook, not a dashboard story with invented graphs or customer names. I wrote down what I tried, what broke, and what I would actually repeat. You can steal the audit and the tests even if you throw my narrative away. I rebuilt the hang locally so every claim here has a command you can run.

Why this hang keeps surviving code review

Do most Python HTTP clients treat a missing timeout as permission to wait forever? Requests does, and urllib.request.urlopen does the same unless a global default exists. That behavior is lovely in a REPL and lethal inside a worker that must finish a job. Ask an assistant for a tiny fetch helper and you often get requests.get(url) with no timeout= keyword at all.

Is that an assistant problem or a documentation problem? It is both, because the generated snippet matches the happy-path examples people copy. The call looks typed, the linter stays quiet, and the job never returns a body. If you are wiring background jobs or agent tool calls in 2026, this is the silent hang you will debug last.

Field notes from the forty-eight hours

Hours 0–8: I blamed the broker first

I started with the queue, because that is where sleeping workers usually hide their secrets. Consumer registration looked fine, heartbeats still landed, and the job sat in a started state without a traceback. I restarted the process twice and watched the same payload refuse to move.

Then I blamed DNS, because a stuck resolver can freeze a process without burning a core. I ran dig and getent hosts against the upstream name and got answers immediately. So why was the worker still smiling with an empty log and a warm heartbeat?

I reached for a stack dump next, expecting a deadlock in our own locks or a forgotten child process. The frames sat inside socket.recv, which I misread as proof the network was merely slow. Have you made that same leap from a quiet stack to a routing ticket?

Hours 8–24: I chased the network and made it worse

I captured packets and found a TCP handshake that completed, then a long polite silence. The remote side had accepted the connection and never sent a response body. Our client had no read deadline, so it waited like a guest who never checks a watch.

I wrapped the call in signal.alarm, which is a terrible idea in a threaded worker, and it seemed to help one run. The next run exploded because another library also wanted SIGALRM for its own deadline. That is when I stopped treating this as a flaky network and started treating it as a missing argument.

I asked a coding assistant to make the request more reliable, and it added retries around the same naked GET. Have you trusted a retry wrapper and assumed that word meant the call could die on its own? Extra attempts without a per-attempt timeout are how one hung socket becomes a small pile of hung sockets.

Hours 24–40: the break was one keyword

I opened the helper the assistant had written on day one and found requests.get(url, headers=headers). I checked the Requests documentation again: when timeout is omitted, the default is None, and None means wait forever. Forever is a long time to wait for a JSON body that will never arrive.

I reproduced it with a local server that accepts a connection and then sleeps. The client process sat in recv until I killed it with a shell. No exception, no warning, no budget, no log line that said we were still waiting. That was the entire bug, and everything before it was theater.

Hours 40–48: what I would repeat without shame

I would start with a grep for HTTP calls that never mention timeout, before I touch the broker UI. I would fail the build if a call site cannot name a number for its budget. I would refuse extra attempts until a single try can die on its own clock. I would keep a silent server in the test suite so the helper cannot regress during a cleanup pull request.

Would I still use an assistant after that mess? Yes, but I would paste the audit output first and ask it to close holes, not invent a new client.

Artifact: a small AST audit for missing timeouts

The script below is a lab linter, not a security scanner and not a proof about production traffic. It walks Python files, looks for common HTTP call patterns, and prints every call site that never names timeout. Dynamic helpers will slip through, which is why the test in the next section exists.

#!/usr/bin/env python3
"""timeout_audit.py — fail if matched HTTP calls omit timeout=."""
from __future__ import annotations

import ast
import sys
from pathlib import Path

HTTP_ATTRS = {"get", "post", "put", "patch", "delete", "head", "request", "send"}
HTTP_NAMES = {"request", "urlopen"}


class TimeoutVisitor(ast.NodeVisitor):
    def __init__(self, filename: str) -> None:
        self.filename = filename
        self.findings: list[str] = []

    def _has_timeout(self, keywords: list[ast.keyword]) -> bool:
        return any(k.arg == "timeout" for k in keywords)

    def visit_Call(self, node: ast.Call) -> None:
        interesting = False
        name = "<call>"
        if isinstance(node.func, ast.Attribute):
            name = node.func.attr
            interesting = name in HTTP_ATTRS
        elif isinstance(node.func, ast.Name):
            name = node.func.id
            interesting = name in HTTP_NAMES

        if interesting and not self._has_timeout(node.keywords):
            self.findings.append(
                f"{self.filename}:{node.lineno}: {name}() has no timeout="
            )
        self.generic_visit(node)


def audit(paths: list[Path]) -> list[str]:
    findings: list[str] = []
    for root in paths:
        files = [root] if root.is_file() else root.rglob("*.py")
        for path in files:
            if path.suffix != ".py":
                continue
            source = path.read_text(encoding="utf-8")
            tree = ast.parse(source, filename=str(path))
            visitor = TimeoutVisitor(str(path))
            visitor.visit(tree)
            findings.extend(visitor.findings)
    return findings


if __name__ == "__main__":
    targets = [Path(p) for p in sys.argv[1:] or ["."]]
    hits = audit(targets)
    for line in hits:
        print(line)
    if hits:
        print(f"\n{len(hits)} call site(s) missing timeout=", file=sys.stderr)
        raise SystemExit(1)
    print("ok: every matched HTTP call names timeout=")

Commands I kept in the notebook:

python timeout_audit.py src tests
rg -n "requests.get|requests.post|urlopen" src
python -m pytest test_http_timeout.py -vv

Does a green audit mean you cannot hang? No, and I need you to hear that clearly. The checker is a tripwire for the helpers assistants regenerate, not a substitute for a transport-level timeout. Label this as a static lab tool: it will miss session.request(method, url, **kwargs) the moment timeout hides inside kwargs.

Artifact: a silent server that fails closed

I wanted a test that hangs the build if someone later "simplifies" the helper back to three lines. Bind a socket, accept one client, and never write a body. The client must raise inside a short budget, not sit there until a human notices pytest has gone quiet.

# test_http_timeout.py
from __future__ import annotations

import socket
import threading
from http.client import HTTPConnection

import pytest


def start_silent_server() -> tuple[str, int, socket.socket]:
    server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    server.bind(("127.0.0.1", 0))
    server.listen(1)
    host, port = server.getsockname()

    def serve() -> None:
        try:
            conn, _ = server.accept()
            # Accept and stall. Do not send a response body.
            threading.Event().wait(timeout=30)
            conn.close()
        except OSError:
            pass

    threading.Thread(target=serve, daemon=True).start()
    return host, port, server


def fetch_with_budget(host: str, port: int, timeout: float) -> None:
    conn = HTTPConnection(host, port, timeout=timeout)
    try:
        conn.request("GET", "/slow")
        conn.getresponse()
    finally:
        conn.close()


def test_client_dies_when_server_stays_silent() -> None:
    host, port, server = start_silent_server()
    try:
        with pytest.raises((TimeoutError, OSError)):
            fetch_with_budget(host, port, timeout=0.3)
    finally:
        server.close()

I used http.client so the notebook does not depend on Requests, httpx, or a vendor SDK. Swap fetch_with_budget for your real helper and keep the silent server. If that test ever fails to return, your helper still inherited timeout=None from a default you never read.

What should you do when pytest itself never comes back? Treat that as a failed test, not as a slow suite, and go read the helper again. A mock that answers in two milliseconds will not save you here.

Decision table I wish I had on hour one

Symptom	First guess I made	What I should have checked	Repeatable action
Worker "running", CPU near zero	Broker deadlock	Stack sitting in socket.recv	Dump the process stack before restarting
One job never finishes	Slow upstream	Client timeout= argument	Grep the helper for timeout
More attempts make it worse	Flaky network	Extra tries without a per-attempt budget	Cap a single attempt first
Assistant "fixed" reliability	Need a sturdier client	New helper still omits timeout	Re-run timeout_audit.py
Tests pass on a laptop	Not a client bug	Mocks that answer immediately	Add the silent-server test

Tape that table next to the helper, because the next assistant will cheerfully regenerate the same three-line client. What number would you put on this call if I asked you right now? If nobody in the file can answer, the worker is already hanging and you have not noticed yet.

Where a coding assistant actually helped

After the audit existed, I wanted a second pass over call sites that might have been styled around the checker. I copied the repo onto a scratch machine, ran the script, and asked an assistant to patch only the listed lines.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I used MonkeyCode for that loop because free model access and a free server option were enough to rerun the audit without parking hung sockets on my laptop. I did not need a model name, a quota claim, or a latency chart. I needed an isolated shell, the test file, and a prompt that said add timeout= and do not invent extra attempts.

Assistants are decent at filling a table you already designed. They are reckless at inventing reliability features you did not ask for. The order matters more than the tool: reproduce the hang, land the tripwire, then let a model touch the helper.

Limitations you should assume are waiting

• The AST check misses session.request(method, url, **kwargs) when timeout hides inside kwargs.
• It will not see generated clients, gRPC stubs, or HTTP spoken from a C extension.
• A client timeout does not save you from a server that drips one byte per minute on purpose.
• timeout=0.3 in the test is not a production budget; take the number from your SLO.
• signal.alarm is not an HTTP timeout, and I am telling you not to copy that detour.
• This notebook has no latency percentiles, error rates, or traffic captures from a real fleet.

Who should skip this notebook

Skip the AST grep if your HTTP layer is entirely generated and a transport already requires a timeout. Skip the silent-server test if your CI image cannot bind 127.0.0.1 or cannot start threads. Skip assistants on the first pass if you cannot read a stack sitting in socket.recv without help.

If you ship money-moving or safety-critical workers, treat this as a checklist, not a control. Put the budget in the adapter, enforce it in review, and keep the silent-server test. Do not run production traffic on a scratch coding box, free or otherwise.

What I would repeat tomorrow

I would still start with one question, asked out loud, before I open the broker UI. Can this call die, and can someone point at the number that kills it? If the answer is a shrug, I would run the audit, land the silent-server test, and only then let an assistant edit the helper. That order is the whole lesson, and it still holds if you throw away every tool I used except the two files above.