48-Hour Field Notes: My Client Timed Out, the Server Finished the Work, and My Batch Ran Twice

Two nights ago my batch job wrote 200 rows and I expected 200 rows. What I got was 261, and nothing in the logs said "error" — just a scattering of timeouts I had trained myself to ignore. So here is the question I should have asked 48 hours earlier: when my HTTP client gives up, who told the server to stop?

Nobody did. That is the whole article, but the interesting part is how quietly it happens, and what a cheap harness looks like that would have caught it before the nightly run.

What the logs actually showed

My worker logged timeout after 3.0s and moved on to the next item, which felt like normal resilience at the time. The remote side, meanwhile, kept generating, finished at roughly 5.5 seconds, and returned a perfectly good result to a socket nobody was reading anymore. My retry then submitted the same logical request a second time, and the writer happily inserted both.

Three details made this hard to see:

1. The failure was classified as failed, not unknown, so the retry policy treated it as safe to repeat.
2. The output table had no unique constraint on the natural key, so duplicates were structurally allowed.
3. Every log line was locally true — my client really did time out — so grepping for errors found nothing wrong.

Isn't that the worst kind of bug? One where every component is behaving exactly as written, and the composition is still wrong.

The two clocks I was measuring as one

A client timeout is a statement about my patience, not about remote state. Once I separated them, the state machine got smaller and the code got easier.

In my harness, a request can end in exactly three ways:

• ok — I read a response body I trust.
• unknown — I stopped waiting; the remote side may have completed, may be mid-flight, or may never have started.
• rejected — the remote side explicitly refused (4xx that is not a rate limit, or a well-formed error payload).

The bug lived in collapsing unknown into failed. failed invites a retry; unknown invites a reconciliation.

The stub that reproduces it in 20 seconds

I refuse to debug timing bugs against a live dependency, so I wrote a stub that is deliberately slower than the client's read timeout. Here it is, unedited and runnable:

# stub_server.py — python stub_server.py, then run the client below
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
import json, threading, time

SEEN = {}
LOCK = threading.Lock()

class Handler(BaseHTTPRequestHandler):
    def do_POST(self):
        n = int(self.headers.get("Content-Length", 0))
        self.rfile.read(n)
        key = self.headers.get("Idempotency-Key", "missing")
        with LOCK:
            first_delivery = key not in SEEN
            SEEN[key] = SEEN.get(key, 0) + 1
            deliveries = SEEN[key]
        if first_delivery:
            time.sleep(3.5)          # longer than the client's 3s budget
        body = json.dumps({"key": key, "deliveries": deliveries}).encode()
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.end_headers()
        self.wfile.write(body)

    def log_message(self, *args):
        pass

ThreadingHTTPServer(("127.0.0.1", 8099), Handler).serve_forever()

And the client, with the classification I should have had from day one:

# client.py — python client.py
import json, time, urllib.request, uuid

BUDGET_S = 3.0

def call(url, payload, key):
    # 80% of the budget for the read; the rest is for serialize + write
    read_timeout = max(1.0, BUDGET_S * 0.8)
    req = urllib.request.Request(
        url,
        data=json.dumps(payload).encode(),
        headers={"Content-Type": "application/json", "Idempotency-Key": key},
    )
    started = time.monotonic()
    try:
        with urllib.request.urlopen(req, timeout=read_timeout) as resp:
            return {"state": "ok", "body": json.loads(resp.read())}
    except TimeoutError:
        # socket.timeout is a TimeoutError subclass on Python 3.10+
        return {"state": "unknown", "elapsed": time.monotonic() - started}

print(call("http://127.0.0.1:8099", {"item": 1}, str(uuid.uuid4())))

Run both, and the client prints {'state': 'unknown'} while the server has already counted one delivery. Run a naive retry loop on top and the stub's counter reaches 2, 3, 4 — the duplicate writes, visible in one integer. My own timings came from my laptop and yours will differ, but the shape of the result does not.

The idempotency key has to be stored before the call

A key generated at request time is useless when the process restarts, so I moved it into the outbox row instead.

CREATE TABLE IF NOT EXISTS outbox (
  key         TEXT PRIMARY KEY,   -- generated once, at row creation
  item_id     TEXT NOT NULL,
  state       TEXT NOT NULL,      -- pending | sent | ok | unknown | rejected
  attempts    INTEGER NOT NULL DEFAULT 0,
  updated_at  TEXT NOT NULL
);

Before calling out, I flip the row to sent and increment attempts; after the call I write ok, rejected, or unknown. A sweeper then resolves unknown rows by either polling a status endpoint or re-sending with the same key, which is safe only if the receiving side deduplicates on it. This is where the practice part bit me twice.

Where I ran the slow path

A stub only proves my own logic, so I also wanted one real remote endpoint with unpredictable latency and no billing setup — which is where I used the free model access and free server option on MonkeyCode (Disclosure: This article was prepared as part of MonkeyCode's product outreach.) It was useful precisely because I do not control its latency: it forced me to measure a p95 before picking a budget instead of guessing one. I am not quoting timings here, because a number I measured on one evening would be a number someone else trusts wrongly.

The decision table I now keep next to the retry code

Observation	Meaning	Action
Client timeout, no response body	Remote state unknown	Reconcile first; re-send only with the same key
429 with Retry-After	Remote side is shedding load	Wait the stated interval; do not double the backoff
5xx after the request was accepted	Partial work possible	Treat as unknown, not failed
4xx validation error	Deterministic rejection	Mark rejected; retrying is pure waste
Two ok responses for one key	Receiver does not dedupe	Fix the write path before touching timeouts

And a rule of thumb for the budget itself: set the read timeout from a measured p95, not from a feeling, and cap total attempts so that attempts × budget still fits inside whatever outer deadline your scheduler enforces.

Limitations, and who should skip this

HTTP has no standard cancellation verb, so a generic "abort" in your client library cancels your wait, not the remote computation. If the receiving side offers no idempotency mechanism and no status lookup, you cannot get exactly-once from this pattern — you can only shrink the window and deduplicate at your write layer.

Skip this approach entirely if:

• your job issues one request per hour, where a manual rerun is cheaper than the outbox;
• you genuinely need hard exactly-once semantics across systems you do not control;
• you cannot change the write path, in which case a unique constraint on the natural key is the only lever you have left.

What I would repeat in the next 48 hours

Before the next batch touches a remote endpoint, I will run the stub, confirm the duplicate counter moves, add the outbox table, and only then look at timeouts. It took me 48 hours to stop reading timeout as failure, and the fix was ten lines of state classification plus one unique index. If you have a nightly job that retries on timeout, the cheapest experiment available to you is the two files above — the counter either stays at 1 or it does not.