48-Hour Field Notes: My 30-Second Timeout Didn't Cancel Anything, and the Server Logged Three Requests

I spent the better part of two days believing that a client timeout meant "the request is over." It doesn't. A timeout is a statement about my patience, not about the work happening on the other side of the socket, and the gap between those two things is where duplicated work, wasted quota, and confusing logs quietly live.

So here is the field note: I built a ten-minute harness that makes the gap visible, then I rewrote my retry loop around what it showed me. If your code posts to a model endpoint and you have ever tuned timeout= by feel, this is the experiment I'd run before you tune it again.

What I Actually Believed

My mental model was a light switch: I send a request, I wait, and if I get impatient I flip the switch off and the other side stops. Three things are being conflated in that sentence, and it took a log to separate them.

1. My deadline — a timer inside my HTTP client. When it fires, my client gives up.
2. The connection close — a TCP FIN or reset on the wire. It tells the peer I'm gone, eventually.
3. Server-side cancellation — the handler deciding to stop computing. This requires the server to cooperate; nothing about my timer forces it.

Only the first one is mine to control. The second is best-effort. The third may never happen at all, because a server can be halfway through a long generation, holding a buffer, and completely uninterested in my impatience.

The Harness: A Server That Ignores You

I didn't want to debug this against a real endpoint, where I can't see the other side. So I wrote a stub server with a hard-coded five-second handler and a client with a one-second timeout. That mismatch is the experiment.

# stub_server.py — a "slow model" that logs receipt and completion separately
import json, threading, time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

LOG, LOCK = [], threading.Lock()

class Handler(BaseHTTPRequestHandler):
    protocol_version = "HTTP/1.1"

    def do_POST(self):
        rid = self.headers.get("X-Request-Id", "missing")
        self.rfile.read(int(self.headers.get("Content-Length", "0")))
        with LOCK:
            LOG.append({"rid": rid, "event": "received", "t": time.time()})
        time.sleep(5)  # pretend this is a long generation
        with LOCK:
            LOG.append({"rid": rid, "event": "finished", "t": time.time()})
        body = json.dumps({"ok": True, "rid": rid}).encode()
        try:
            self.send_response(200)
            self.send_header("Content-Type", "application/json")
            self.send_header("Content-Length", str(len(body)))
            self.end_headers()
            self.wfile.write(body)
        except (BrokenPipeError, ConnectionResetError):
            with LOCK:
                LOG.append({"rid": rid, "event": "client_gone", "t": time.time()})

    def do_GET(self):
        with LOCK:
            body = json.dumps(LOG).encode()
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def log_message(self, *args):
        pass

ThreadingHTTPServer(("127.0.0.1", 8765), Handler).serve_forever()

# client.py — three attempts, one-second deadline, same payload
import json, time, urllib.request

for attempt in range(3):
    rid = f"req-{attempt}"
    req = urllib.request.Request(
        "http://127.0.0.1:8765/generate",
        data=json.dumps({"prompt": "hello"}).encode(),
        headers={"Content-Type": "application/json", "X-Request-Id": rid},
        method="POST",
    )
    try:
        with urllib.request.urlopen(req, timeout=1) as r:
            print(rid, "ok", r.status)
    except Exception as e:
        print(rid, "gave up:", type(e).__name__)
    time.sleep(0.5)

Run the server in one terminal, the client in another, then curl -s localhost:8765/log a few seconds later.

Reading the Log Was the Whole Lesson

The log came back with nine entries, not three, and the ordering was the interesting part:

Event	Count in one run	What it means
received	3	Three requests reached the handler
client_gone	3	My connection dropped each time
finished	3	The work completed anyway

Three retries, three completions, zero successful responses on my side. If that handler had been a paid model endpoint, I would have been charged for three generations and kept the output of none of them. My "retry" wasn't a retry at all — it was a second and third unit of work.

And notice the subtlety that broke my first theory: client_gone did fire. The server knew I left. It just didn't stop, because there was nothing in the handler that checks whether the caller is still listening. That check has to exist, and on many APIs it doesn't, or it isn't exposed to you.

The question I now ask first

Before adding a retry, I ask: can I tell, after the fact, whether the previous attempt landed? If the answer is no, the retry is a guess wearing a safety vest.

The Fix: An Intent Ledger, Not a Retry Counter

I replaced the loop with two ideas. First, every attempt carries a stable idempotency key derived from the job, not from the attempt number, so a cooperating server can dedupe. Second, I write an intent record to disk before the call and resolve it after — which is exactly what makes the fix survive a process restart.

import hashlib, json, pathlib, urllib.request

LEDGER = pathlib.Path("ledger")
LEDGER.mkdir(exist_ok=True)

def job_key(payload: dict) -> str:
    raw = json.dumps(payload, sort_keys=True).encode()
    return hashlib.sha256(raw).hexdigest()[:16]

def call_once(payload: dict, deadline: float) -> dict | None:
    key = job_key(payload)
    marker = LEDGER / f"{key}.intent"
    if (LEDGER / f"{key}.done").exists():
        return json.loads((LEDGER / f"{key}.done").read_text())
    if not marker.exists():
        marker.write_text(json.dumps({"payload": payload, "attempts": 0}))
    state = json.loads(marker.read_text())
    if state["attempts"] >= 2:
        raise RuntimeError(f"{key}: unresolved after 2 attempts, needs a human")
    state["attempts"] += 1
    marker.write_text(json.dumps(state))

    req = urllib.request.Request(
        "http://127.0.0.1:8765/generate",
        data=json.dumps(payload).encode(),
        headers={"Content-Type": "application/json", "X-Request-Id": key},
        method="POST",
    )
    try:
        with urllib.request.urlopen(req, timeout=deadline) as r:
            result = json.loads(r.read())
    except Exception:
        return None  # unresolved: the intent marker stays, no silent infinite retry
    (LEDGER / f"{key}.done").write_text(json.dumps(result))
    return result

The .done and .intent split is deliberately boring. Files survive a reboot, they're greppable, and when I find a stray .intent with no .done on a Monday morning I know precisely which job I have to re-check by hand instead of guessing.

Decision Table I Now Keep Next to the Code

Situation on timeout	Action	Reason
No bytes received, server supports an idempotency key	Retry once with the same key	The server can dedupe the duplicate
No bytes received, no idempotency support	Cap at two attempts, then mark unresolved	Beyond two, cost grows and certainty doesn't
Partial stream received	Don't blind-retry; discard and record	You cannot cleanly resume a half-parsed answer
Long batch on a disposable box	Write .intent before the call	The box may vanish mid-flight
Deadline under 2× expected latency	Raise the deadline before adding retries	Retries on a too-short deadline are self-inflicted load

That last row is the one I got wrong for a day. I had a 30-second deadline on calls that routinely took longer, so every "failure" was really a measurement error, and the retry loop was just amplifying it.

Where the Free Tier Actually Helped, and Where It Didn't

I reran the harness against a real endpoint because the stub can only teach semantics, not behaviour. MonkeyCode's free model access and free server option were enough to do that without provisioning anything: the harness, the stub, and the client all ran in one place, and I could watch the ledger grow while the calls were in flight. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What I did not verify, and won't claim: whether the upstream endpoint cancels generation when a client disconnects, what its idle or total timeouts are, or whether it dedupes on a key. Those are provider-specific behaviours I'd want a status page or docs page for before I put numbers next to them. My harness tells you what your client did — that's the part you own.

Limitations worth stating plainly

The ledger assumes a single writer per key; two workers on the same job would race on the .intent file. A database row with a unique constraint is the upgrade path. Streaming endpoints need separate handling because a partial answer is not a retryable unit. And none of this helps if your deadline is shorter than the p99 latency you actually observe — fix the deadline first.

Who should not use this approach

Skip it if your calls are idempotent reads with no cost per attempt, if your provider exposes a documented request-status API you can poll instead, or if you're prototyping and genuinely don't care about duplicate work. The ledger is for jobs where a duplicate is expensive or where an unresolved job must be found later, not for every request you'll ever make.

What I'd Repeat in the Next 48 Hours

• Run the stub harness before touching a retry policy. Ten minutes, no dependencies, and it changes how you read your own logs.
• Log receipt and completion as separate events on both sides whenever you can. One event hides everything.
• Give each job a stable key, and let the key — not the attempt counter — be what the server sees.
• Print the count of .intent files with no .done in your daily check. It's a one-line alert that catches the failures retries used to hide.

If you take one thing from these notes, take the question: when my client times out, who actually stops working? Write the harness once, and stop guessing at your timeout config.