DEV Community: Taylor Wang

Two Duplicate Rows in 48 Hours: My Unique Index Treats NULL as a New Value

Taylor Wang — Tue, 27 Oct 2026 22:09:57 +0000

I shipped a nightly enrichment job with a unique index on (tenant_id, source_hash) and called it idempotent.
Forty-eight hours later I had three rows for one tenant and one source document, all identical, and the index had approved every one of them.
If a unique index is unique, how does an insert get past it twice? The answer is not a race, a bug in Postgres, or a failed commit — it is one nullable column and a rule most of us read once and then forget.

I want to write this down because the fix took ten minutes and the diagnosis took a weekend, and because the harness I used is short enough to paste into any throwaway database.

What the job was supposed to guarantee

The contract was small and, I thought, airtight:

One row per (tenant_id, source_hash) pair, forever.
Re-runs are safe: if the row exists, skip the expensive step.
A skipped row must never mean a half-written row.

The expensive step was a model call that labelled each document, so skipping mattered. My reasoning was that a unique index plus ON CONFLICT DO NOTHING gives me exactly-once semantics for free. That reasoning holds right up until one of the indexed columns is NULL.

The two rows that should not exist

Here is the query that ended my weekend, run against the staging copy:

SELECT tenant_id, source_hash, count(*)
  FROM enrichment
 GROUP BY tenant_id, source_hash
HAVING count(*) > 1;

One row came back: acme | NULL | 3. Not a hash collision, not a duplicate tenant — a NULL.

Two documents had failed to hash: one fetch returned an empty body, one returned a redirect page I had not handled. My loader wrote source_hash = NULL, shrugged, and continued. Every subsequent retry did the same thing, and every retry counted as a brand-new row.

Why the database happily said yes

In SQL, a UNIQUE constraint is satisfied when any indexed column is NULL, because NULL is not equal to anything — including another NULL. Postgres follows this rule for single and multi-column unique indexes alike, and MySQL and SQLite behave the same way for the common cases.

So my index was doing precisely what I asked: ('acme', NULL) and ('acme', NULL) are distinct values as far as it is concerned. ON CONFLICT (tenant_id, source_hash) DO NOTHING never fired, because no conflict was ever detected.

The uncomfortable part is that NULL was the error state. The rows that most needed deduplication were the only rows exempt from it.

A reproduction you can run in five minutes

I ran this against a throwaway Postgres instance; the free server option from MonkeyCode is enough for the sandbox, since nothing here needs tuning or size. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Step 1 — the schema I actually shipped

CREATE TABLE enrichment (
  id          bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  tenant_id   text        NOT NULL,
  source_hash text,                    -- nullable: this is the door
  model_label text,
  payload     jsonb       NOT NULL,
  created_at  timestamptz NOT NULL DEFAULT now()
);

CREATE UNIQUE INDEX enrichment_dedupe_v1
  ON enrichment (tenant_id, source_hash);

Step 2 — the test that proves the index does nothing

BEGIN;

INSERT INTO enrichment (tenant_id, source_hash, payload)
VALUES ('acme', NULL, '{"doc":"a"}');

INSERT INTO enrichment (tenant_id, source_hash, payload)
VALUES ('acme', NULL, '{"doc":"a"}')
ON CONFLICT (tenant_id, source_hash) DO NOTHING;

SELECT count(*) AS rows_for_acme FROM enrichment WHERE tenant_id = 'acme';
ROLLBACK;

Expected output on a stock Postgres: 2. If you were expecting 1, you have now reproduced my entire weekend in one transaction.

Step 3 — confirm the same insert is blocked when the hash is present

BEGIN;
INSERT INTO enrichment (tenant_id, source_hash, payload)
VALUES ('acme', 'sha256:aaa', '{"doc":"b"}'),
       ('acme', 'sha256:aaa', '{"doc":"b"}')
ON CONFLICT (tenant_id, source_hash) DO NOTHING;
SELECT count(*) FROM enrichment WHERE tenant_id = 'acme';  -- 1
ROLLBACK;

That contrast is the whole lesson: the index works, the data model was wrong.

Fixes, ranked by how much they change

Option	Works when	Cost	Gotcha
`NOT NULL` + a real sentinel hash	You can always compute a hash	Backfill + `ALTER COLUMN`	You must decide what an unhashable document means
`UNIQUE NULLS NOT DISTINCT` (Postgres 15+)	You want the DB to treat `NULL` as a value	One index rebuild	Behaviour differs from every other index you own
Expression index on `COALESCE(source_hash, '<none>')`	You are on an older Postgres	No app change to the column	`ON CONFLICT` must repeat the exact expression
Two partial unique indexes	`NULL` rows need a different key	Two indexes to maintain	Predicate must appear in `ON CONFLICT`
App-level advisory lock	Genuinely cannot change schema	Slower, easier to leak	Hides the modelling problem

I chose the first option, then added the second as belt and braces, because a nightly job that silently duplicates output is worse than a nightly job that fails loudly. The loader now writes into a separate enrichment_rejected table when a document cannot be hashed, which turns a silent NULL into a visible queue with an owner.

-- Option 1: make the column lie-proof
UPDATE enrichment
   SET source_hash = 'unhashable:' || md5(payload::text)
 WHERE source_hash IS NULL;

ALTER TABLE enrichment ALTER COLUMN source_hash SET NOT NULL;

-- Option 2, on Postgres 15+
DROP INDEX enrichment_dedupe_v1;
CREATE UNIQUE INDEX enrichment_dedupe_v2
  ON enrichment (tenant_id, source_hash) NULLS NOT DISTINCT;

Clean up before you tighten the index

Do not add the constraint first. It will fail on your existing duplicates, and if you build it with CREATE INDEX CONCURRENTLY the failure arrives minutes later and leaves an invalid index behind.

WITH ranked AS (
  SELECT id,
         row_number() OVER (
           PARTITION BY tenant_id, source_hash
           ORDER BY created_at, id
         ) AS rn
    FROM enrichment
   WHERE source_hash IS NOT NULL
)
SELECT count(*) AS duplicate_groups FROM ranked WHERE rn > 1;

Then run the same shape without the WHERE, partitioned by (tenant_id, md5(payload::text)), to find the NULL clusters the first query deliberately skips.

The `ON CONFLICT` trap I hit on attempt two

My first attempt at a fix used an expression index, and my inserts kept duplicating anyway. Arbiter inference is literal: ON CONFLICT (tenant_id, source_hash) cannot use an index built on COALESCE(source_hash, '<none>'), so Postgres fell back to reporting no conflict.

The insert has to name the same expression the index names, character for character, or the conflict target will not match. It reads like a duplicate of the DDL and that is exactly the point.

INSERT INTO enrichment (tenant_id, source_hash, payload)
VALUES ('acme', NULL, '{"doc":"a"}')
ON CONFLICT (tenant_id, COALESCE(source_hash, '<none>')) DO NOTHING;

Who should not take this route

Teams where NULL genuinely means "unknown and possibly different". Collapsing all unknowns into one key is data loss with extra steps.
Writers you do not control. If another service inserts into the table, tightening the column is a contract change, not a migration.
Very large tables during business hours. A unique index build is a full scan; plan the window and expect the duplicate check to be the slow part.
Anyone who needs cross-database portability. NULLS NOT DISTINCT is Postgres-specific, so the sentinel-hash fix travels better.

What I would repeat in the next 48 hours

Write the negative test before shipping the index — two NULL inserts in one transaction, assert count = 1, done in thirty seconds.
Treat every error path as a data path, because the loader's sad path wrote rows too.
Separate "cannot process" from "processed with no value" in the schema, not in a comment.
Grep for ON CONFLICT after any index change, since the arbiter is a string match and not a semantic one.

If you want to try the harness, the SQL above is the entire test and a free-tier Postgres is sufficient; paste it, watch the count come back as 2, then decide which row in the table should have been impossible.

MonkeyCode provides free models that can run this workflow.

48-Hour Field Notes: My Latency Numbers Were Wrong Because I Started the Clock in the Wrong Place

Taylor Wang — Tue, 27 Oct 2026 19:04:45 +0000

For two days I tried to answer one question about a free model endpoint and a free server box: is this combination fast enough to keep? I answered it wrong, and not because the box was slow. I answered it wrong because I put the starting line of my stopwatch in the wrong place, then trusted the number that fell out of it.

Here is the part that stings a little. A wrong clock does not produce an obviously wrong number; it produces a confident, well-formatted number that survives review because nobody asks where start came from. So this post is not a review of any provider. It is the harness I rebuilt on the second pass, the queries that make the mistake visible, and the decision rule I now use before I judge any free tier.

The three numbers I actually needed

When you run a batch job against a model endpoint on a box you do not own, "how fast is it" is not one number. It is at least three, and they answer different questions:

Queue wait — how long an item sits inside my own process before a request leaves the machine. This measures my semaphore, my thread pool, my scheduler. It says nothing about the provider.
Service time — how long the request itself takes, from send to full body. This is the only slice the provider controls.
Attempts per item — how many requests I spent to get one usable answer. This is the multiplier that quietly eats your rate budget.

If you collapse all three into one elapsed_ms field, you get a number that moves when any of them moves, and you cannot tell which one moved. So which of those three do you think your current logs actually separate out?

The mistake: one `time.time()` for four different intervals

My first pass looked like this, and it looked reasonable at the time:

import time, requests

SESSION = requests.Session()

def classify(text: str):
    start = time.time()
    resp = SESSION.post(ENDPOINT, json=payload(text), timeout=90)
    elapsed_ms = (time.time() - start) * 1000
    log.info("classify", extra={"elapsed_ms": elapsed_ms, "status": resp.status_code})
    return resp.json()

This single timer hides four separate things, and each one is enough to flip a build-or-abandon decision:

Connection setup is inside the measurement. DNS, TCP, and TLS happen before the model sees anything, so a cold pool looks like a slow model.
Client-side retries multiply rows. Three attempts on one item produce three elapsed_ms samples, so your average is now an average over attempts, not over items.
Wall clock can step. On a box you do not control, an NTP correction can move time.time() mid-request, and the delta can even come out negative.
Queue wait is invisible. If items wait on a semaphore for forty seconds, that wait never appears anywhere, because the timer starts after the wait.

Notice that none of these are provider bugs. They are all my instrumentation, which is exactly why they are worth writing down.

Rebuild: one row per attempt, four timestamps

On the second pass I logged one row per attempt instead of one row per call, and I kept two kinds of time on purpose. Durations come from time.monotonic() inside the process, because monotonic cannot step. Wall timestamps are stored too, but only to join and order rows across processes, never to compute a duration.

CREATE TABLE IF NOT EXISTS attempt (
  attempt_id     TEXT PRIMARY KEY,
  item_id        TEXT NOT NULL,
  try_index      INTEGER NOT NULL,
  queued_wall    REAL NOT NULL,
  sent_wall      REAL,
  done_wall      REAL,
  queue_wait_ms  INTEGER NOT NULL,
  service_ms     INTEGER NOT NULL,
  status         INTEGER,
  outcome        TEXT NOT NULL,   -- ok | retryable | permanent
  error          TEXT
);

CREATE INDEX IF NOT EXISTS idx_attempt_item ON attempt(item_id, try_index);

The writer is boring on purpose, because instrumentation that can throw is instrumentation that will throw at 3am:

import sqlite3, time, uuid, requests

DB = sqlite3.connect("attempts.sqlite3")
SESSION = requests.Session()
RETRYABLE = {408, 429, 500, 502, 503, 504}

def attempt(item_id, payload, try_index, url, headers):
    queued_wall, queued = time.time(), time.monotonic()
    resp, sent = None, None
    try:
        resp = SESSION.post(url, json=payload, headers=headers, timeout=(5, 90))
        sent = time.monotonic()
        outcome = "ok" if resp.status_code == 200 else (
            "retryable" if resp.status_code in RETRYABLE else "permanent")
        error = None
    except requests.RequestException as exc:
        sent = time.monotonic()
        outcome, error = "retryable", type(exc).__name__
    done = time.monotonic()
    DB.execute(
        "INSERT INTO attempt (attempt_id, item_id, try_index, queued_wall, sent_wall,"
        " done_wall, queue_wait_ms, service_ms, status, outcome, error)"
        " VALUES (?,?,?,?,?,?,?,?,?,?,?)",
        (str(uuid.uuid4()), item_id, try_index, queued_wall, time.time(), time.time(),
         int((sent - queued) * 1000), int((done - sent) * 1000),
         getattr(resp, "status_code", None), outcome, error))
    return resp, outcome

One honest limitation: this does not split DNS and TLS apart from generation, because requests does not hand you that boundary without a transport hook. For my decision it did not matter, since both are client-side and I control the pool. If yours are slow, reach for a hook before you reach for a bigger box.

The three queries that make the mistake obvious

Logs you never query are just disk usage, so the harness is only half the artifact. These three queries are what I run before forming any opinion.

-- 1. Retry amplification: requests spent per item answered
SELECT COUNT(*) AS attempts,
       COUNT(DISTINCT item_id) AS items,
       ROUND(1.0 * COUNT(*) / COUNT(DISTINCT item_id), 2) AS attempts_per_item
FROM attempt;

-- 2. Split the wall time: my backpressure vs the endpoint
SELECT ROUND(AVG(queue_wait_ms), 1) AS avg_queue_wait_ms,
       ROUND(AVG(service_ms), 1)    AS avg_service_ms,
       ROUND(SUM(queue_wait_ms) * 1.0
             / SUM(queue_wait_ms + service_ms), 3) AS queue_share
FROM attempt WHERE outcome = 'ok';

-- 3. Per-item latency, not per-attempt latency
SELECT ROUND(AVG(total_ms), 1) AS avg_item_ms FROM (
  SELECT item_id, SUM(queue_wait_ms + service_ms) AS total_ms
  FROM attempt WHERE outcome = 'ok' GROUP BY item_id
);

Run query 3 without the GROUP BY, and you are averaging retries into your latency. That number trends upward as the endpoint gets flakier, which makes a provider problem look like a speed problem and sends you optimizing the wrong layer.

Read the shape, not the score

The output is only meaningful as a shape, so I stopped writing percentages on a whiteboard and started matching shapes to actions. The rows below are labels, not measurements you can borrow; the numbers in any sample output you generate are yours.

Shape in your log	Likely cause	What I do next
`queue_share` high, `service_ms` flat	My own concurrency limit or worker starvation	Raise workers, shrink payloads, stop blaming the endpoint
`service_ms` noisy, `attempts_per_item` near 1.0	Endpoint variance, no retry pressure	Keep it for batch work, keep it out of interactive paths
`attempts_per_item` above 1.5	Timeouts or rejections being retried	Fix timeout budget and backoff before touching limits
`outcome = 'permanent'` clustered	Bad payloads being retried pointlessly	Classify the error before the retry, every time

That fourth row deserves a paragraph. A client that retries anything non-200 will happily burn your entire budget on a malformed request, and it will look busy rather than broken. Classify first, then retry, and write permanent to the table so the next person can see it.

Where MonkeyCode's free model access and free server fit in this workflow

I ran the collector and the job loop on MonkeyCode's free server option and pointed the client at its free model access, which changed two things about the experiment. First, instrumentation multiplies request count, and when calls are free there is no billing conversation about re-running a batch to check variance. Second, a box I do not own forces me to treat the clock and the disk as unreliable inputs, which is the correct default anyway; I would have skipped that discipline on hardware I trusted.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What I will not do here is quote throughput or latency figures as if they were a property of the platform. Availability is the claim, and the harness above is how you turn availability into a decision for your own workload, on your own machine, with your own numbers.

What broke, and what I would repeat

Field notes should be honest about the boring failures, so here is the list from my two days:

SQLite writes became the bottleneck once workers went up. I batched inserts in memory and flushed every 200 rows; the log stopped competing with the job.
A missing index made my analysis query slower than the workload, which is a funny way to discover you forgot idx_attempt_item.
time.time() bit me once with a negative delta after a clock correction, which is why durations are monotonic-only now.
I logged the payload twice in the first version, once in the row and once in a JSON blob, and the log grew faster than the results.

What I would repeat without hesitation is the ordering: instrument first, decide second. It costs one extra half hour and saves you from writing a confident postmortem about the wrong component.

Who should not use this approach

Anyone who needs billing-grade accounting. Provider-side usage records are the authority there, not my local rows.
Interactive products with a hard per-request budget. You need distributed tracing with spans, not a single SQLite file on one box.
Teams running fewer than a few hundred calls. Variance dominates the sample, and you will read tea leaves instead of shapes.
Anyone who will not run the queries. A log nobody aggregates is a slower job with extra disk I/O.

The 20-minute test plan I now run first

Point the client at the endpoint with a payload you already trust, and cap it at 200 items.
Wrap every call in attempt() so each request becomes one row, including retries.
Run once with concurrency 1, then again at the concurrency you plan to ship.
Run queries 1 and 2 and compare the two runs, paying attention to queue_share moving while service_ms stays put.
Add one deliberately malformed payload and confirm it lands as permanent instead of being retried.
Only then look at query 3 and decide whether the box earns a longer trial.

The whole thing fits in a single file plus a SQLite database, and it survives being run against anything that speaks HTTP.

The question I should have asked on day one

If a number moves when you change your own concurrency limit, is it really telling you anything about the endpoint? Mine was not, and I spent two days reading my own scheduler's fingerprints as if they were a provider's. Instrumenting the boundaries takes an afternoon; arguing about a latency figure with no boundaries takes longer and ends in a guess.

If you want somewhere to run the harness while you tune it, MonkeyCode's free model access is a reasonable place to point the probe, and its free server option is enough to host the collector; just keep the 20-minute test plan, because the harness is the part that travels with you.

48-Hour Field Notes: My 30-Second Timeout Didn't Cancel Anything, and the Server Logged Three Requests

Taylor Wang — Tue, 27 Oct 2026 15:59:24 +0000

I spent the better part of two days believing that a client timeout meant "the request is over." It doesn't. A timeout is a statement about my patience, not about the work happening on the other side of the socket, and the gap between those two things is where duplicated work, wasted quota, and confusing logs quietly live.

So here is the field note: I built a ten-minute harness that makes the gap visible, then I rewrote my retry loop around what it showed me. If your code posts to a model endpoint and you have ever tuned timeout= by feel, this is the experiment I'd run before you tune it again.

What I Actually Believed

My mental model was a light switch: I send a request, I wait, and if I get impatient I flip the switch off and the other side stops. Three things are being conflated in that sentence, and it took a log to separate them.

My deadline — a timer inside my HTTP client. When it fires, my client gives up.
The connection close — a TCP FIN or reset on the wire. It tells the peer I'm gone, eventually.
Server-side cancellation — the handler deciding to stop computing. This requires the server to cooperate; nothing about my timer forces it.

Only the first one is mine to control. The second is best-effort. The third may never happen at all, because a server can be halfway through a long generation, holding a buffer, and completely uninterested in my impatience.

The Harness: A Server That Ignores You

I didn't want to debug this against a real endpoint, where I can't see the other side. So I wrote a stub server with a hard-coded five-second handler and a client with a one-second timeout. That mismatch is the experiment.

# stub_server.py — a "slow model" that logs receipt and completion separately
import json, threading, time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

LOG, LOCK = [], threading.Lock()

class Handler(BaseHTTPRequestHandler):
    protocol_version = "HTTP/1.1"

    def do_POST(self):
        rid = self.headers.get("X-Request-Id", "missing")
        self.rfile.read(int(self.headers.get("Content-Length", "0")))
        with LOCK:
            LOG.append({"rid": rid, "event": "received", "t": time.time()})
        time.sleep(5)  # pretend this is a long generation
        with LOCK:
            LOG.append({"rid": rid, "event": "finished", "t": time.time()})
        body = json.dumps({"ok": True, "rid": rid}).encode()
        try:
            self.send_response(200)
            self.send_header("Content-Type", "application/json")
            self.send_header("Content-Length", str(len(body)))
            self.end_headers()
            self.wfile.write(body)
        except (BrokenPipeError, ConnectionResetError):
            with LOCK:
                LOG.append({"rid": rid, "event": "client_gone", "t": time.time()})

    def do_GET(self):
        with LOCK:
            body = json.dumps(LOG).encode()
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def log_message(self, *args):
        pass

ThreadingHTTPServer(("127.0.0.1", 8765), Handler).serve_forever()

# client.py — three attempts, one-second deadline, same payload
import json, time, urllib.request

for attempt in range(3):
    rid = f"req-{attempt}"
    req = urllib.request.Request(
        "http://127.0.0.1:8765/generate",
        data=json.dumps({"prompt": "hello"}).encode(),
        headers={"Content-Type": "application/json", "X-Request-Id": rid},
        method="POST",
    )
    try:
        with urllib.request.urlopen(req, timeout=1) as r:
            print(rid, "ok", r.status)
    except Exception as e:
        print(rid, "gave up:", type(e).__name__)
    time.sleep(0.5)

Run the server in one terminal, the client in another, then curl -s localhost:8765/log a few seconds later.

Reading the Log Was the Whole Lesson

The log came back with nine entries, not three, and the ordering was the interesting part:

Event	Count in one run	What it means
`received`	3	Three requests reached the handler
`client_gone`	3	My connection dropped each time
`finished`	3	The work completed anyway

Three retries, three completions, zero successful responses on my side. If that handler had been a paid model endpoint, I would have been charged for three generations and kept the output of none of them. My "retry" wasn't a retry at all — it was a second and third unit of work.

And notice the subtlety that broke my first theory: client_gone did fire. The server knew I left. It just didn't stop, because there was nothing in the handler that checks whether the caller is still listening. That check has to exist, and on many APIs it doesn't, or it isn't exposed to you.

The question I now ask first

Before adding a retry, I ask: can I tell, after the fact, whether the previous attempt landed? If the answer is no, the retry is a guess wearing a safety vest.

The Fix: An Intent Ledger, Not a Retry Counter

I replaced the loop with two ideas. First, every attempt carries a stable idempotency key derived from the job, not from the attempt number, so a cooperating server can dedupe. Second, I write an intent record to disk before the call and resolve it after — which is exactly what makes the fix survive a process restart.

import hashlib, json, pathlib, urllib.request

LEDGER = pathlib.Path("ledger")
LEDGER.mkdir(exist_ok=True)

def job_key(payload: dict) -> str:
    raw = json.dumps(payload, sort_keys=True).encode()
    return hashlib.sha256(raw).hexdigest()[:16]

def call_once(payload: dict, deadline: float) -> dict | None:
    key = job_key(payload)
    marker = LEDGER / f"{key}.intent"
    if (LEDGER / f"{key}.done").exists():
        return json.loads((LEDGER / f"{key}.done").read_text())
    if not marker.exists():
        marker.write_text(json.dumps({"payload": payload, "attempts": 0}))
    state = json.loads(marker.read_text())
    if state["attempts"] >= 2:
        raise RuntimeError(f"{key}: unresolved after 2 attempts, needs a human")
    state["attempts"] += 1
    marker.write_text(json.dumps(state))

    req = urllib.request.Request(
        "http://127.0.0.1:8765/generate",
        data=json.dumps(payload).encode(),
        headers={"Content-Type": "application/json", "X-Request-Id": key},
        method="POST",
    )
    try:
        with urllib.request.urlopen(req, timeout=deadline) as r:
            result = json.loads(r.read())
    except Exception:
        return None  # unresolved: the intent marker stays, no silent infinite retry
    (LEDGER / f"{key}.done").write_text(json.dumps(result))
    return result

The .done and .intent split is deliberately boring. Files survive a reboot, they're greppable, and when I find a stray .intent with no .done on a Monday morning I know precisely which job I have to re-check by hand instead of guessing.

Decision Table I Now Keep Next to the Code

Situation on timeout	Action	Reason
No bytes received, server supports an idempotency key	Retry once with the same key	The server can dedupe the duplicate
No bytes received, no idempotency support	Cap at two attempts, then mark unresolved	Beyond two, cost grows and certainty doesn't
Partial stream received	Don't blind-retry; discard and record	You cannot cleanly resume a half-parsed answer
Long batch on a disposable box	Write `.intent` before the call	The box may vanish mid-flight
Deadline under 2× expected latency	Raise the deadline before adding retries	Retries on a too-short deadline are self-inflicted load

That last row is the one I got wrong for a day. I had a 30-second deadline on calls that routinely took longer, so every "failure" was really a measurement error, and the retry loop was just amplifying it.

Where the Free Tier Actually Helped, and Where It Didn't

I reran the harness against a real endpoint because the stub can only teach semantics, not behaviour. MonkeyCode's free model access and free server option were enough to do that without provisioning anything: the harness, the stub, and the client all ran in one place, and I could watch the ledger grow while the calls were in flight. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What I did not verify, and won't claim: whether the upstream endpoint cancels generation when a client disconnects, what its idle or total timeouts are, or whether it dedupes on a key. Those are provider-specific behaviours I'd want a status page or docs page for before I put numbers next to them. My harness tells you what your client did — that's the part you own.

Limitations worth stating plainly

The ledger assumes a single writer per key; two workers on the same job would race on the .intent file. A database row with a unique constraint is the upgrade path. Streaming endpoints need separate handling because a partial answer is not a retryable unit. And none of this helps if your deadline is shorter than the p99 latency you actually observe — fix the deadline first.

Who should not use this approach

Skip it if your calls are idempotent reads with no cost per attempt, if your provider exposes a documented request-status API you can poll instead, or if you're prototyping and genuinely don't care about duplicate work. The ledger is for jobs where a duplicate is expensive or where an unresolved job must be found later, not for every request you'll ever make.

What I'd Repeat in the Next 48 Hours

Run the stub harness before touching a retry policy. Ten minutes, no dependencies, and it changes how you read your own logs.
Log receipt and completion as separate events on both sides whenever you can. One event hides everything.
Give each job a stable key, and let the key — not the attempt counter — be what the server sees.
Print the count of .intent files with no .done in your daily check. It's a one-line alert that catches the failures retries used to hide.

If you take one thing from these notes, take the question: when my client times out, who actually stops working? Write the harness once, and stop guessing at your timeout config.

48-Hour Field Notes: My Homemade Eval Set Was Scoring My Prompt, Not the Model

Taylor Wang — Tue, 27 Oct 2026 13:55:14 +0000

I spent two days comparing models on my own little eval set, and the winner changed every time I reordered the examples. That is not a model difference, that is my eval set leaking.

Here is the setup. Free model access means I can afford three repeats per item instead of one, and a free server option means the sweep can run overnight without me babysitting a laptop. Cheap inference makes homemade evals attractive. But cheap inference also makes it easy to build an eval that measures nothing.

These are my field notes: what I tried, what broke, and the two scripts I would keep.

What I thought I was measuring

I had 60 labelled items for a boring extraction task: pull three fields out of a short support message and return JSON. I wrote eight few-shot examples, picked four candidate models, and ran each item once.

Three numbers came out of that first sweep, and only one of them was about correctness:

Format compliance — did the reply parse as JSON with the three keys present?
Abstention behaviour — when the message did not contain a field, did the model say so or invent a value?
Agreement — did the extracted value match my label?

Agreement is the number everyone quotes. Format compliance and abstention are the ones that page you at 2am. And all three were lying to me, because of a bug that had nothing to do with any model.

The bug: my prompt examples were inside my eval

I built the eval set first, then wrote few-shot examples by copying the clearest items. That is the whole bug. Five of my eight examples were near-duplicates of eval items, so any model that copied an example verbatim got a free point.

You can spot this by eye at 60 items. You cannot spot it at 600, and you definitely cannot spot it after you have paraphrased your prompt three times and forgotten what moved.

So I stopped trusting my eyes and wrote a checker.

Artifact 1: a shingle overlap gate for prompt/eval leakage

The check is deliberately dumb: tokenise, build k-word shingles, and fail when any eval item shares too large a fraction of its shingles with the prompt examples. No embeddings, no LLM judge, nothing you have to trust.

#!/usr/bin/env python3
"""leak_check.py - fail when eval items share too much text with prompt examples.

Usage:
    python leak_check.py prompt_examples.json eval_set.json --threshold 0.30
Exit code 1 means the eval set is contaminated: fix the split before trusting a score.
"""
import argparse
import json
import re
import sys
from pathlib import Path

TOKEN = re.compile(r"[a-z0-9]+")


def shingles(text: str, k: int = 5) -> set[str]:
    words = TOKEN.findall(text.lower())
    if not words:
        return set()
    if len(words) < k:
        return {" ".join(words)}
    return {" ".join(words[i:i + k]) for i in range(len(words) - k + 1)}


def load_items(path: str) -> list[dict]:
    data = json.loads(Path(path).read_text(encoding="utf-8"))
    return data["items"] if isinstance(data, dict) else data


def main() -> int:
    ap = argparse.ArgumentParser()
    ap.add_argument("prompt_examples")
    ap.add_argument("eval_set")
    ap.add_argument("--threshold", type=float, default=0.30)
    ap.add_argument("--k", type=int, default=5)
    args = ap.parse_args()

    prompt_grams: set[str] = set()
    for item in load_items(args.prompt_examples):
        prompt_grams |= shingles(item["text"], args.k)

    bad = []
    for item in load_items(args.eval_set):
        grams = shingles(item["text"], args.k)
        if not grams:
            continue
        hit = grams & prompt_grams
        ratio = len(hit) / len(grams)
        if ratio >= args.threshold:
            bad.append((item["id"], ratio, sorted(hit)[:2]))

    for item_id, ratio, sample in sorted(bad, key=lambda row: -row[1]):
        print(f"CONTAMINATED {item_id}: {ratio:.2%} overlap, e.g. {sample}")
    print(f"{len(bad)} flagged item(s) at k={args.k}, threshold={args.threshold:.0%}")
    return 1 if bad else 0


if __name__ == "__main__":
    sys.exit(main())

Two rules made this useful in practice. First, run it in CI on every prompt change, not once at design time. Second, when an item trips the gate, rewrite the item instead of deleting it, so your coverage does not quietly shrink every time the gate fires.

The second bug: one sample per item is a coin flip

After the leakage fix, agreement numbers dropped and got noisier. Why? Because I was reading a single sample per item and treating it as a property of the model.

Extraction tasks are boring until the message is ambiguous, and ambiguity is exactly where sampling temperature does its work. A model that is "88% correct" might be getting the same 12% wrong every time, or a different 12% each run. Those are different products.

So I changed the unit of measurement from accuracy to flip rate: the share of items whose label changed across three repeats of the same input.

# sweep.py - N repeats per item per model, one JSONL row per call.
import json
import time
import uuid


def sweep(model_id, items, repeats=3, out="runs.jsonl"):
    with open(out, "a", encoding="utf-8") as fh:
        for item in items:
            for r in range(repeats):
                row = {"model": model_id, "item": item["id"], "repeat": r, "call": str(uuid.uuid4())}
                t0 = time.monotonic()
                try:
                    text = call_model(model_id, item["text"])   # your client, 1 concurrent call
                    row["label"] = parse_label(text)
                    row["error"] = None
                except Exception as exc:
                    row["label"] = None
                    row["error"] = type(exc).__name__
                row["ms"] = round((time.monotonic() - t0) * 1000)
                fh.write(json.dumps(row) + "\n")
                fh.flush()   # a restart must not eat finished work

Summarising is six lines, and it tells you more than the accuracy table did:

import collections, json

rows = [json.loads(line) for line in open("runs.jsonl", encoding="utf-8")]
labels = collections.defaultdict(set)
errors = collections.Counter()
for r in rows:
    if r["error"]:
        errors[(r["model"], r["error"])] += 1
    else:
        labels[(r["model"], r["item"])].add(r["label"])

for model in sorted({m for m, _ in labels}):
    sets = [v for (m, _), v in labels.items() if m == model]
    flipped = sum(1 for v in sets if len(v) > 1)
    print(f"{model}: {flipped}/{len(sets)} items flipped across repeats")
print(dict(errors))

A high flip rate on a free model is not a reason to reject it. It is a reason to know which items are unstable and to route those to a human or a second pass.

Where free access and a free server actually fit

I use MonkeyCode's free model access for the candidate sweep and its free server option to keep the sweep running when my laptop sleeps. That is the honest role: the free tier buys me repeats, which is the only thing that made flip rate measurable at all. If inference cost per call were the constraint, I would have run one sample per item and shipped the noisy table.

Capabilities change, so treat availability as something to verify on the provider's current terms rather than a fixed promise. I deliberately do not quote quotas or model names here, because a number that was true when I wrote this is exactly the kind of stale figure that makes an eval post useless.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The tool is not the point. Swap it for anything and the scripts still work.

What broke, and what I would repeat

What broke:

Prompt examples copied from eval items. Fixed with the shingle gate, not with discipline.
Single-sample scoring. Fixed by repeats and flip rate.
A crash mid-sweep. The flush() per row and the call UUID saved me; without them I could not tell duplicates from retries.
Silently changing the prompt mid-sweep. Now the sweep writes the prompt hash into every row.

What I would repeat:

Write the holdout after the prompt is frozen, never before.
Keep the gate in CI so future me cannot "just quickly" copy an example into the eval.
Compare format compliance and abstention before accuracy, because a model that returns prose cannot be parsed and a model that never abstains cannot be trusted.

A comparison table that survives price being zero

When every candidate is free, cost stops being the tiebreaker, so replace it with stability:

Signal	What it catches	When it outweighs accuracy
Format compliance	Unparseable replies	Always, for pipelines
Abstention rate	Confident invention	Fields that drive money or state
Flip rate across repeats	Non-determinism	Anything you cannot re-run later
Error mix by type	Timeouts vs. truncation vs. refusals	When you plan retries
Latency p50	Queue pressure only	Last, as a tiebreaker

Limitations, and who should skip this

Sixty items is not a benchmark. The gate catches copy-paste leakage, not paraphrase leakage, so a human-written twin of an example will sail through. Flip rate with three repeats gives you a smell, not a confidence interval, and none of this replaces a real labelled sample of production traffic.

Skip this approach if you need contractual uptime or a data-processing agreement you have not read, or if your inputs cannot leave your boundary at all — a hosted free tier is the wrong shape for that, no matter how good the numbers look.

If you already have a large frozen eval set, you probably do not need the shingle gate; you need a better sampling policy. If you have no eval set yet, that is the actual first problem.

So: when you next see a headline jump on a narrow task, what would you check first — the model, or the set you scored it on? Run the gate on your own prompt before you answer; it takes about a minute and it will probably find something.

48-Hour Field Notes: I Ramped Worker Threads From 1 to 12 and Watched Throughput Fall Off a Cliff at 8

Taylor Wang — Tue, 27 Oct 2026 12:54:31 +0000

I had roughly 400 short model calls to make and a free server that I did not pay for, so the obvious move was to crank up the thread count and walk away. Twenty minutes in, my script was faster on paper and slower on wall clock, and I had no measurement that could tell me which of those two facts was real. Have you ever tuned a number because it felt faster?

These are my 48-hour field notes from that week: what I tried, what broke, and the one small harness I would build again before writing any job queue. Everything below is from a single box, a single run, and a free model endpoint — treat the numbers as an observation, not a benchmark of anything.

What I was actually optimizing

The job was boring on purpose: for each of ~400 inputs, ask a model for a short structured summary, then append the result to a file. The per-call latency was the whole cost — my Python did almost no work. So the only lever I thought I had was concurrency.

Serial run: roughly 8 minutes, which felt wasteful.
12 threads: finished, but slower than serial on one of the three runs.
Retries: silently multiplied the request count, so my "throughput" was fiction.

That last bullet is the trap. If you count successes per second without counting attempts, concurrency looks free right up until it does not.

The product side, briefly

I was running this on MonkeyCode's free server option and calling its free model access, which is why per-call latency was worth measuring at all rather than just paying for headroom. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I did not get internal queue metrics, so every claim below is something I observed from the client side with a stopwatch.

Attempt 1: more threads, less thinking

My first version was the classic "just raise the pool" edit:

from concurrent.futures import ThreadPoolExecutor
with ThreadPoolExecutor(max_workers=12) as pool:
    results = list(pool.map(call_model, inputs))

It is four lines and it hides everything I needed. There is no per-call deadline, no record of attempts, no separation between "the endpoint was slow" and "my box was busy", and the results only exist after the last call returns. When the host recycling happened at hour 31 and the process died, I had nothing to analyze.

So the fix was not a better pool size. The fix was instrumentation first.

The harness I should have written on day one

Ninety lines, standard library only, no dependencies. It sends N calls at a given concurrency, records every attempt as one JSON line, and prints a small summary. You can paste this and change the two constants.

import json, time, urllib.request, urllib.error
from concurrent.futures import ThreadPoolExecutor

URL = "http://127.0.0.1:8000/v1/chat/completions"  # your local/forwarded endpoint
PAYLOAD = {"messages": [{"role": "user", "content": "Reply with the single word: ok"}]}

def one_call(deadline_s=30):
    req = urllib.request.Request(
        URL,
        data=json.dumps(PAYLOAD).encode(),
        headers={"Content-Type": "application/json"},
    )
    t0 = time.monotonic()
    try:
        with urllib.request.urlopen(req, timeout=deadline_s) as r:
            r.read()
            status = r.status
        return {"ok": True, "ms": round((time.monotonic() - t0) * 1000), "status": status}
    except urllib.error.HTTPError as e:
        return {"ok": False, "ms": round((time.monotonic() - t0) * 1000), "status": e.code}
    except Exception as e:
        return {"ok": False, "ms": round((time.monotonic() - t0) * 1000), "status": type(e).__name__}

def pct(xs, p):
    xs = sorted(xs)
    if not xs:
        return None
    k = max(0, min(len(xs) - 1, int(round((p / 100) * (len(xs) - 1)))))
    return xs[k]

def sweep(concurrency, total=40):
    t0 = time.monotonic()
    with ThreadPoolExecutor(max_workers=concurrency) as pool:
        rows = list(pool.map(lambda _: one_call(), range(total)))
    wall = time.monotonic() - t0
    lat = [r["ms"] for r in rows]
    oks = [r for r in rows if r["ok"]]
    print(json.dumps({
        "conc": concurrency,
        "rps": round(len(oks) / wall, 2),
        "p50_ms": pct(lat, 50), "p95_ms": pct(lat, 95),
        "errors": len(rows) - len(oks),
    }))

for c in (1, 2, 4, 8, 12):
    sweep(c)

Two details matter more than the code. First, rps is computed from successful calls only, so a failing endpoint cannot look fast. Second, every attempt is timed individually, so a p95 that triples while p50 barely moves tells you something is queueing — and it is probably not your CPU.

What one sweep showed

One run, one free box, 40 calls per step, all replies identical. This is a single observation, not a product benchmark, and I would not quote it as capacity.

Workers	ok req/s	p50 (ms)	p95 (ms)	errors
1	0.8	1,180	1,900	0
2	1.5	1,240	2,300	0
4	2.6	1,410	3,100	0
8	3.0	2,050	6,800	1
12	2.7	3,600	11,400	6

Read it as two curves, not one. Throughput flattens between 4 and 8; tail latency keeps climbing past 8 and errors appear. The knee is where p95 grows faster than req/s, and for this box on that afternoon it was somewhere around 4 to 6.

Is that number stable? Almost certainly not. It depends on time of day, on whatever else the free host is doing, and on how long my prompts are. That is exactly why the harness matters more than the number.

What broke, and what the failure was not

My first theory was CPU saturation on the free server. Wrong. When I logged the box's own load while the sweep ran, it stayed low — the waiting was happening on the network hop and at the endpoint, not in my process. The retries were the real damage: every timeout re-sent a request that the endpoint was probably still working on, which made the next sweep look worse and pushed me toward even more threads. Does that loop sound familiar?

Three concrete changes came out of it.

Fixed pool, explicit queue. Four workers, a bounded queue, and a deadline_s short enough that a stuck call dies before the next batch starts.
Attempt-level logging. One JSON line per attempt with conc, ms, ok, status, and a run id. Re-analysis never requires re-running.
Warm-up before measuring. The first two calls after an idle period were consistently the slowest, so I throw them away instead of letting them pollute p50.

After that, the 400-call run landed near 3 minutes and, more importantly, it was predictable. I cared less about the ceiling than about not being surprised by it.

Decision table

Your situation	Do this	Skip
One-off script, <100 calls	Serial, log everything	Concurrency tuning
Recurring batch on a free box	Sweep 1→12 once per day, pick the knee	Copying someone else's worker count
Latency-sensitive user requests	Measure p95 per concurrency step first	Retrying aggressively on timeout
Hard quota or per-call cost	Count attempts, not successes	Optimizing req/s in isolation

Who should not use this approach

The honest limits: my sweep is 40 calls per step on a shared free box, with no control over neighboring load, so it cannot tell you what capacity you will get tomorrow. If you need a guaranteed p95, contractual throughput, or spend accounting at the request level, a free tier and a home-grown harness are the wrong foundation — measure, then move to a paid plan with published limits. And do not run any of this against data you would not send to a third party.

What I would repeat, and what I wasted time on

I would absolutely repeat the sweep and the per-attempt JSONL log; together they turned a vague "it feels slow" into a table I could argue with. I would skip the day I spent reading about thread pool tuning before I had a single measurement of my own, because the bottleneck was never where the blog posts said it would be.

If you take one thing: make the endpoint prove its speed to you at the concurrency you actually plan to run, and write that proof down before you trust it. If you try the sweep, change URL and total and post your table — I would like to know whether your knee sits where mine did.

48-Hour Field Notes: I Crash-Tested My Resumable Batch Job Before Trusting a Free Box

Taylor Wang — Tue, 27 Oct 2026 09:40:15 +0000

The first version of my batch pipeline ran end to end on the first attempt, and that is exactly why I distrusted it. A job that finishes once tells you nothing about what it does at item 37, when the process disappears and nobody is watching. So I spent 48 hours on the least glamorous work I know: build a resumable queue, then try hard to kill it.

These are field notes, in the format I keep coming back to because it forces me to write down the boring parts. Here is what I tried, what broke, and what I would repeat.

The setup, and the constraint that shaped it

The workload was small on purpose: a few dozen items, each one a model call plus a short transform, all written to disk as JSON. I ran it on the free server option and sent the calls through MonkeyCode's free model access, because the whole experiment had to be cheap enough to run badly five times.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The expensive part was never the model. It was my assumption that a long-running process on a machine I do not own will live long enough to finish. Ask yourself honestly: how many of your batch jobs would survive a restart at a random moment?

The resume contract: three states, one fenced write

I stopped thinking in terms of the job runs and started thinking in terms of a contract. Every item moves pending -> claimed -> done, and every transition is one transaction with a guard clause, so a stale worker cannot commit work it no longer owns.

The rules I settled on:

A worker may only claim items that are pending, or claimed with an expired lease.
A worker may only write done if it still holds the lease for that item, enforced inside the WHERE clause itself.
Side effects are keyed by item_id (out/{item_id}.json), so redoing work is wasteful but never duplicating.
Attempts are counted on claim, not on failure, because a killed process never gets to report a failure.

Rule 4 is the one I got wrong first, and it cost me most of an afternoon.

# ledger.py (stdlib only)
import os, sqlite3, time

DB = os.environ.get('LEDGER_DB', 'ledger.sqlite3')
LEASE = float(os.environ.get('LEASE_SECONDS', '90'))

def connect():
    con = sqlite3.connect(DB, isolation_level=None, timeout=30)
    con.execute('PRAGMA journal_mode=WAL')
    con.execute('''CREATE TABLE IF NOT EXISTS items (
        item_id TEXT PRIMARY KEY,
        state TEXT NOT NULL CHECK (state IN ('pending','claimed','done')),
        attempts INTEGER NOT NULL DEFAULT 0,
        lease_until REAL,
        worker TEXT,
        output TEXT)''')
    return con

def claim(con, worker, limit=1):
    con.execute('BEGIN IMMEDIATE')
    try:
        now = time.time()
        rows = con.execute('''SELECT item_id FROM items
            WHERE state = 'pending'
               OR (state = 'claimed' AND lease_until < ?)
            ORDER BY item_id LIMIT ?''', (now, limit)).fetchall()
        for (item_id,) in rows:
            con.execute('''UPDATE items
                SET state = 'claimed', attempts = attempts + 1,
                    lease_until = ?, worker = ?
                WHERE item_id = ?''', (now + LEASE, worker, item_id))
        con.execute('COMMIT')
        return [r[0] for r in rows]
    except Exception:
        con.execute('ROLLBACK')
        raise

def complete(con, item_id, worker, output_path):
    con.execute('BEGIN IMMEDIATE')
    try:
        cur = con.execute('''UPDATE items
            SET state = 'done', output = ?, lease_until = NULL
            WHERE item_id = ? AND state = 'claimed' AND worker = ?''',
            (output_path, item_id, worker))
        if cur.rowcount != 1:
            con.execute('ROLLBACK')
            raise RuntimeError('lease lost for ' + item_id)
        con.execute('COMMIT')
    except Exception:
        raise

That worker = ? inside the guard is the whole trick. Without it, a process that wakes up after its lease expired will cheerfully overwrite the result of the worker that took over. Have you ever written that UPDATE without the fence? I had, for years, and it never showed up until a machine restarted under me.

Crash injection: making failure reproducible

Real crashes are rare, which makes them a terrible test input. So I added a hook that kills the process at a chosen item, using os._exit so no finally block, no atexit handler, and no buffer flush gets a chance to run.

# worker.py (core loop)
import os, json, uuid
from ledger import connect, claim, complete

WORKER = uuid.uuid4().hex[:8]
CRASH_AT = int(os.environ.get('CRASH_AT', '0'))
con = connect()
done_here = 0

while True:
    ids = claim(con, WORKER, limit=1)
    if not ids:
        break
    item_id = ids[0]
    text = 'result for ' + item_id        # stands in for the model call
    with open(os.path.join('out', item_id + '.json'), 'w') as f:
        json.dump({'item_id': item_id, 'text': text}, f)
    complete(con, item_id, WORKER, 'out/' + item_id + '.json')
    done_here += 1
    if CRASH_AT and done_here == CRASH_AT:
        os._exit(9)                        # closest thing to kill -9

Then the test plan, which is the part I would keep forever:

python seed.py 50
for n in 0 1 7 19 33 49; do
  CRASH_AT=$n python worker.py
  python verify.py || echo "FAILED at CRASH_AT=$n"
done
python worker.py            # no hook: drain the queue
python verify.py

CRASH_AT=0 means no crash, so the last pass is a normal drain. If your ledger is correct, the queue finishes no matter where the earlier runs were killed, and the total attempts stay bounded.

verify.py checks three things, and each of them caught something real:

# verify.py
import os, sqlite3, sys
con = sqlite3.connect('ledger.sqlite3')
rows = con.execute('SELECT item_id, state, attempts FROM items').fetchall()
not_done = [r for r in rows if r[1] != 'done']
missing  = [r[0] for r in rows if not os.path.exists('out/' + r[0] + '.json')]
thrashing = [r for r in rows if r[2] > 3]
for label, bad in (('not done', not_done), ('missing output', missing), ('too many attempts', thrashing)):
    print(label + ':', bad)
sys.exit(1 if (not_done or missing or thrashing) else 0)

The too-many-attempts check is not about correctness; it is about whether my lease is shorter than my slowest item. If it fires, I tune the lease before I tune anything else.

What broke in the first 24 hours

The claim and the write were not in one transaction. My first version claimed in autocommit mode and wrote done in a second statement, so a kill between them left items claimed with no output. The fix was BEGIN IMMEDIATE around each transition, plus attempts counted on claim.

Expired claims were invisible until I looked for them. A killed worker leaves state='claimed' forever if nothing re-eligible it. My query said WHERE state = 'pending', so the queue looked empty while half the items sat in limbo. Adding lease expiry to the claim query made the queue self-heal.

Progress counted the wrong thing. I printed SELECT COUNT(*) FROM items and called it progress, which hit one hundred percent the moment I seeded the table. Counting state='done' is the only number that means anything to a human waiting on a run.

A file write outside the transaction. Writing out/x.json before the done update is safe only because the filename is keyed. The moment I appended to a shared results.jsonl instead, a restarted worker appended the same item twice. If your sink cannot be keyed, you need a dedupe table, not optimism.

When the free box is the right place for this

Situation	Free server option	A box you control
Batch work restartable at any item	Good fit	Also fine
Unkeyed external side effects	No, add dedupe first	No
One long job with no checkpoints	No	No
Deadline-critical production run	No	Only with monitoring
Learning or measurement runs	Good fit	Fine

The pattern is simple: the free option is a good fit when losing the machine is cheap. It is a bad fit when losing the machine is the incident.

Limitations I will not pretend away

SQLite is a single-writer store. Fine for one box, wrong for many workers over a network filesystem.
Lease tuning stays guesswork until you measure. Too short and you double-process; too long and a crash costs you a full lease window of idle time.
This gives you at-least-once with idempotent writes. It does not give you exactly-once, and nothing here is billing-grade accounting.
If each item takes 200 milliseconds, none of this is worth the code. Restarting from zero is cheaper than the ledger.
A resume contract protects you from wasted work, not from a loop that never terminates. That needs a separate spend or attempt cap.

And if your pipeline already runs inside a managed scheduler that retries and deduplicates for you, stop reading and go use it. Rebuilding this is only rational when you own neither the scheduler nor the hardware.

What I would repeat

Write the crash hook before the happy path. The happy path is the easy half and you already know it works.
Count attempts on claim, never on exception. Killed processes do not file bug reports.
Fence every terminal write with the worker identity, every single time, even when it feels redundant.
Make verify.py exit non-zero, so the loop above fails loudly instead of printing sad output into a scrollback nobody reads.
Run the worker under setsid so a dropped SSH session is not accidentally part of the test.

If you want to rehearse this on hardware you do not own, MonkeyCode's free server option and free model access are enough to run the exact test plan above. The test plan is the part that transfers, whichever machine you end up on.

48-Hour Field Notes: My Queue Drained to Zero and the Work Wasn't Done

Taylor Wang — Tue, 27 Oct 2026 06:34:55 +0000

What does "done" mean when the process that writes your results can disappear at any second? I spent two days on a batch extraction job that printed a clean zero in the pending column while the results table underneath it was quietly short a few rows. The job called a model I reached through MonkeyCode's free model access, and the worker itself ran on the free server option that comes with the account. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The disclosure is the least interesting part of this post, so let me get to the bug. I had two separate writes per item, no transaction wrapping them, and a counter that read the wrong one of the two. If you have ever watched a queue hit zero and felt relief before checking the output, this story is for you.

Hour 0-6: the counter that lied

My worker was a plain loop: claim an item, call the model, write the payload, mark the item done. The pending count came from the item table, and the item table was always the last thing I touched. So when the box recycled underneath me, the counter was the most accurate-looking number in the system and the least trustworthy one.

Here is the shape of what I had, and yes, it is as thin as it looks:

item = db.claim_next()          # status -> 'done' happens inside here
payload = call_model(item)
db.save_result(item.id, payload)

Can you see the hole? If the process dies between claim_next() and save_result(), the item is permanently marked done with nothing behind it. There is no retry path, no error, no alarm. The queue drains, the dashboard turns green, and the missing rows ship to whoever reads the export next.

Hour 6-18: two writes, no transaction

The fix is not clever. It is the boring one: make completion a single atomic act that includes the result. I moved everything into SQLite with WAL mode enabled and a busy timeout, because the free box had enough memory for that and I did not want to babysit a separate database.

CREATE TABLE items (
  id            TEXT PRIMARY KEY,
  input_hash    TEXT NOT NULL,
  prompt_version TEXT NOT NULL,
  status        TEXT NOT NULL DEFAULT 'pending',
  lease_owner   TEXT,
  lease_expires_at REAL
);

CREATE TABLE results (
  idem_key   TEXT PRIMARY KEY,
  item_id    TEXT NOT NULL,
  payload    TEXT NOT NULL,
  created_at REAL NOT NULL
);

The write became one transaction: insert the result with INSERT OR IGNORE, then flip the item to done. If the insert conflicts, I already have the answer and the flip is a no-op. If the process dies mid-transaction, SQLite rolls back and the item stays claimable. That single change removed the silent-loss failure mode entirely.

The claim needs a lease, not a flag

A boolean in_progress flag is a promise you make to yourself and break on the next restart. I replaced it with a lease: a claim sets an owner and an expiry, and the claim query is happy to steal expired leases back.

CLAIM = """
UPDATE items
   SET status='claimed', lease_owner=?, lease_expires_at=?
 WHERE id IN (
   SELECT id FROM items
    WHERE status='pending'
       OR (status='claimed' AND lease_expires_at < ?)
    ORDER BY id LIMIT ?
 )
RETURNING id, input_hash, prompt_version;
"""

The RETURNING clause needs SQLite 3.35 or newer, which is worth checking before you copy this into an old base image. A heartbeat thread extends the expiry every few seconds while the model call is in flight, so a long call does not get stolen out from under a healthy worker.

Hour 18-30: the restart duplicated calls

With leases in place, my restarts stopped losing work and started duplicating it. The same item came back, the model answered again, and because generation is not deterministic, I got two different payloads for one input. That was worse than the original bug in one specific way: my agreement check compared samples, so duplicated-but-divergent answers made the quality gate flap between pass and fail across runs.

This is the part people skip. Exactly-once delivery is not something you get from a queue; it is something you get from an idempotency key that the worker derives itself. Mine is a hash of the input plus the prompt version, not a random UUID handed out at claim time, because a random key from a retried worker gives you two keys and two rows.

idem_key = sha256(f"{input_hash}:{prompt_version}".encode()).hexdigest()

Hour 30-42: the key needed the prompt version

I originally hashed only the input, and that cost me an afternoon. I edited the extraction prompt, re-ran the batch, and every item hit the cache and returned the old payload without a single model call. The job looked fast because it was doing nothing.

Putting prompt_version in the key means a prompt edit invalidates old rows on purpose, while a plain retry after a crash still hits the same key and reuses the stored answer. If you want the opposite behavior, hash the input alone and version your results table instead — just decide deliberately rather than discovering it at 2 a.m.

Hour 42-48: the kill test I would repeat

The only test that mattered was the one where I murdered the worker mid-call. Not a graceful shutdown, not a timeout, a hard SIGKILL at a random moment, repeated until I stopped being surprised.

# 20 rounds: start, kill -9 after a random delay, then assert invariants
for i in $(seq 1 20); do
  python worker.py & pid=$!
  sleep $(awk -v s=$RANDOM 'BEGIN{printf "%.2f", s/32768}')
  kill -9 $pid; wait $pid 2>/dev/null
  python check_invariants.py --db batch.sqlite || { echo "round $i failed"; exit 1; }
done
echo "all rounds clean"

The invariant script checks three things and nothing else: no item is done without a matching row in results; no idem_key appears twice; every claimed item either has a live lease or has been returned to pending. Write the assertions in plain SQL and print the offending rows, because a bare assert False in a loop tells you nothing about which invariant you broke.

A decision table for the free path

Availability claims like "free models" and "free server" are not the same as a guarantee, and I had to be honest with myself about where each one fits. This is the table I now fill in before starting a batch job:

Situation	Free server + free models?	Why
Overnight batch, results read tomorrow	Yes	Restarts only cost time, and the lease makes time recoverable
Anything a user is waiting on, sub-second	No	You cannot promise a latency budget you do not control
Data you are not allowed to send off-box	No	Check your obligations before the first request, not after
Experiments where the model choice may change	Yes	Prompt and model version belong in the idempotency key anyway
Production writes to a system of record	Not directly	Land results in your own store, then promote deliberately

Notice that none of those rows are about price. The free tier was not the interesting variable; the interesting variable was whether my code could survive a restart without lying to me.

Who should not use this approach

If you need an uptime commitment, a fixed concurrency ceiling, or a compliance story about where inference happens, this workflow is the wrong tool and no amount of retry logic fixes that. If your team will not run the kill test in CI, do not adopt the lease pattern either — an untested lease is just a more complex flag. And if your batch is small enough to finish in one sitting with a human watching, the plain loop is fine; the overhead only pays for itself once restarts are routine.

What I would repeat on Monday

I would keep the leased claim, the single-transaction completion, the prompt version inside the idempotency key, and the 20-round kill loop in CI. I would drop the custom dashboard counter entirely and read completion from the results table, because the counter was never the source of truth and I kept treating it like one.

The uncomfortable lesson is that my first version looked correct for hours at a time, which is exactly long enough to trust it. If you want a place to rehearse that failure on purpose, the free model access and free server option on MonkeyCode are where I ran mine — bring the kill test with you, and tell me which invariant broke first.

48-Hour Field Notes: My Prompt Budget Was in Characters, and the Overflow Never Raised

Taylor Wang — Tue, 27 Oct 2026 03:29:53 +0000

Silent truncation is the worst kind of bug in a batch job, because the run exits zero and the output still looks like prose. I spent two days chasing one that never raised an exception, never logged a warning, and only showed up as answers that stopped mid-thought. What would you blame first — the model, the network, or your own length math? In my case it was the third one, and the fix was smaller than the investigation.

The budget I thought I had

Most of us do the same thing when we write our first batch harness: we count characters, divide by some number we half-remember, and call it a token budget. That number usually comes from English prose, where a rough four-characters-per-token rule feels close enough for a while. Then the corpus changes, the ratio quietly stops holding, and the guard you trusted keeps saying yes.

Here is the shape of the guard I started with. It is short, it is honest about being an estimate, and it was wrong in a way I could not see from the logs:

# budget_guard.py
from dataclasses import dataclass


class BudgetError(RuntimeError):
    """Raised before a request is sent, never after."""


@dataclass(frozen=True)
class PromptBudget:
    max_tokens: int = 800
    chars_per_token: float = 3.6  # tuned on English prose, which was the mistake

    def estimated_tokens(self, prompt: str) -> int:
        return int(len(prompt) / self.chars_per_token) + 1

    def check(self, prompt: str, label: str = "prompt") -> None:
        est = self.estimated_tokens(prompt)
        if est > self.max_tokens:
            raise BudgetError(
                f"{label}: {len(prompt)} chars, ~{est} est tokens, budget {self.max_tokens}"
            )

Calling it is deliberately boring, so that a failure points at the prompt and not at the plumbing:

budget = PromptBudget(max_tokens=800)
for item in batch:
    budget.check(item.prompt, label=item.id)
    send(item.prompt)

Hour 0-6: build the fixture corpus before the pipeline

The first thing I did right was refuse to test with one prompt. I wrote four fixture families — English prose, CJK prose, a base64 blob, and an emoji-heavy log line — because those are the four shapes that actually showed up in my data. Each family is generated by repetition so the test is fully reproducible and needs no network access at all.

# tests/test_budget_guard.py
import pytest
from budget_guard import PromptBudget, BudgetError

BUDGET = PromptBudget(max_tokens=800)

FIXTURES = {
    "english_prose": "the quick brown fox " * 120,      # 2,400 chars
    "cjk_prose": "这是一个用于测试的句子。" * 150,          # 1,950 chars
    "base64_blob": "QUJDREVGR0hJSktMTU5PUFFSU1RVVldYWVo=" * 60,
    "emoji_log": "✅ shipped 🚀 " * 200,
}


def test_english_passes_and_that_is_correct():
    BUDGET.check(FIXTURES["english_prose"], label="english")


def test_cjk_passes_the_guard_and_that_is_the_bug():
    prompt = FIXTURES["cjk_prose"]
    BUDGET.check(prompt, label="cjk")        # does not raise
    assert BUDGET.estimated_tokens(prompt) < 800

Run it with python -m pytest tests/test_budget_guard.py -q and both tests pass on day one. The second one is a test that documents the defect instead of catching it, and I wrote it that way on purpose so the next person sees the blind spot in the suite rather than in production.

Hour 6-20: what actually broke

On many tokenizers, CJK text costs closer to one token per character than to one token per four characters, so a 1,950-character prompt is nearer 1,950 tokens than the 543 my guard estimated. Base64 and other dense alphabets behave in a similar direction, because they carry almost no redundancy for the tokenizer to compress. English prose was the only fixture family where my ratio was roughly right, and it was also the only family I had calibrated against.

The overflow never raised, because the guard only ever saw characters.
The request went out, the endpoint accepted it, and generation simply stopped early for some items.
My downstream validator rejected those outputs as malformed rather than truncated, which sent me looking at schema code for hours.
Because retries looked identical to first attempts in the logs, I could not tell which failures were repeats and which were new.

I was running these batches on free model access through MonkeyCode, so a wasted call cost me wall-clock time rather than money, which is exactly why the bug survived a full day of observation. Free tiers make waste cheap to ignore and expensive to find later. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Hour 20-34: fail loudly, then truncate on purpose

The fix was not a better ratio. It was admitting that a character count is a guess, and that a guess needs a worst-case bound whenever the corpus is mixed. For anything I cannot classify ahead of time, I set chars_per_token=1.0, which over-blocks English and under-blocks nothing I have seen. For known-English corpora I keep 3.6 and accept the risk, because the alternative blocks half my legitimate work.

When the guard fires, I want one of two paths, never a third silent one:

Split the input at a sentence or record boundary and send two calls.
Truncate deliberately with a visible marker, and log the byte offsets you dropped.

BUDGET_STRICT = PromptBudget(max_tokens=800, chars_per_token=1.0)

for item in batch:
    try:
        BUDGET_STRICT.check(item.prompt, label=item.id)
    except BudgetError as exc:
        log.warning("split required: %s", exc)
        for chunk in split_on_boundary(item.prompt, limit=700):
            send(chunk)
        continue
    send(item.prompt)

If your endpoint returns usage counts, calibrate the ratio from them instead of trusting mine, and recalibrate whenever the corpus changes shape. If it does not return counts, treat the ratio as a safety margin and never as a measurement.

A decision table I actually use

Corpus shape	Ratio to set	Guard behaviour	Watch out for
English prose only	3.6	blocks near the limit	drifting toward logs or code
Mixed or unknown	1.0	blocks early, often	throughput loss you must accept
CJK-dominant	1.0	blocks near the real cost	punctuation-heavy text
Base64 / IDs / hashes	1.0	blocks early	silent mid-blob splits
Code with long strings	2.5	middle ground	indentation and quotes inflate counts

Hour 34-48: what I would repeat and what I would skip

What I would repeat: writing the fixture corpus before the pipeline, and keeping one test that documents a known false negative instead of pretending the guard is exact. What I would skip: trying to find the perfect global ratio, which does not exist for a mixed corpus and cost me most of day two.

Limitations and who should not use this

If your SDK exposes the real tokenizer, use it and skip the estimate entirely, because nothing here beats an exact count. If your prompts are homogeneous and short, the guard adds ceremony without much benefit. And if your job can tolerate partial outputs with a downstream repair step, a strict pre-flight block may cost more throughput than it saves in debugging.

A character budget is a guess wearing a number's clothing, and guesses should fail loudly at the boundary rather than quietly inside the response. If you want to run these fixtures without paying per call while you calibrate, the free model access and free server option in MonkeyCode are enough for a harness this small — start with the pytest file above and change one ratio at a time.

48-Hour Field Notes: I Batched Two Items Per Call and Lost Track of Which Answer Belonged to Which Input

Taylor Wang — Tue, 27 Oct 2026 00:24:23 +0000

Halfway through a two-day batch run I noticed that one answer looked suspiciously like the answer to the item two rows above it, and I could not prove which input it came from. Has that ever happened to you — the job finished, the output looked plausible, and you had no way to defend a single row of it? These are my field notes from fixing that, with the harness I ended up keeping.

What I was actually trying to do

I had a flat list of short, independent text items, and I wanted one short answer per item, written to a ledger I could re-run safely. Sending one request per item is the obvious path, but it doubles the number of round trips, and every round trip is another chance to miss a wall-clock deadline. So I tried the cheaper-looking option: put two items in one prompt and ask for two answers back. Note that this is not a quota argument — I never measured a quota, and I am not claiming one exists. The motivation was purely fewer round trips and fewer half-finished batches.

For context, the driver ran on MonkeyCode's free server option, and the calls went through its free model access. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I will keep the product out of the technical claims, because the failure below is a property of my prompt and my validator, not of any provider.

Day one: the harness I wrote

The whole thing is one file with no dependencies beyond the Python standard library, which was deliberate. If the harness can move between machines with scp and nothing else, I can debug it anywhere, including on a machine I do not own. Here is the shape of it.

# batch_harness.py — stdlib only; you supply call_model()
import hashlib
import json
from dataclasses import dataclass, field


def item_id(text: str) -> str:
    return hashlib.sha256(text.strip().encode("utf-8")).hexdigest()[:12]


@dataclass
class Item:
    text: str
    id: str = field(init=False)

    def __post_init__(self):
        self.id = item_id(self.text)


def build_prompt(items):
    blocks = "\n".join(f'<item id="{i.id}">{i.text}</item>' for i in items)
    return (
        "Answer each item separately. Return a JSON array only, no prose.\n"
        'Each element must be {"id": "<id from the item tag>", "answer": "<answer>"}.\n'
        "Copy ids exactly. Do not renumber, merge, or reorder the items.\n\n"
        + blocks
    )

The important line is the id, and it is deliberately boring: a hash of the input text, truncated. Same input, same id, forever — so a second run of the same ledger is a no-op by construction. If the id were derived from the model's output, a reworded answer would look like a brand-new item, and the queue would quietly grow.

The validator, which is where the actual work lives

def validate(items, raw):
    try:
        payload = json.loads(raw)
    except json.JSONDecodeError as exc:
        return None, f"not_json: {exc.msg}"
    if not isinstance(payload, list):
        return None, "not_a_list"

    rows = [r for r in payload if isinstance(r, dict)]
    got = [str(r.get("id", "")).strip().lower() for r in rows]
    want = [i.id for i in items]

    if len(got) != len(set(got)):
        return None, "duplicate_ids"
    if set(got) != set(want):
        missing = sorted(set(want) - set(got))
        extra = sorted(set(got) - set(want))
        return None, f"set_mismatch missing={missing} extra={extra}"

    answers = {r["id"].strip().lower(): r.get("answer", "") for r in rows}
    if any(not isinstance(a, str) or not a.strip() for a in answers.values()):
        return None, "empty_answer"
    return answers, "ok"

Two details I will defend. First, I lowercase and strip the id before comparing, because that transform is lossless — an uppercase hex string identifies exactly one item. Second, I refuse every lossy match: no positional mapping, no prefix matching, no edit distance. Fuzzy matching is precisely how you end up with two answers for one item and a warm feeling that everything is fine.

What broke on day two

Two failure modes showed up, and both were invisible to a try/except around the HTTP call.

Answers came back without ids. The response was a clean JSON array of {"answer": ...} objects, in a plausible order. Nothing errored. A positional mapper would have accepted this and been right — probably.
Ids came back that I never sent. One batch returned eleven-character ids instead of twelve; another returned ids that were valid hex but belonged to nothing in my prompt. Set equality caught both instantly; a length check would not have.

Why is partial acceptance worse than a loud failure here? Because a batch that is 50% valid does not stop the run — it just moves two items into the wrong bucket, and you discover it when someone reads the output.

The fix: fail the batch, then fall back to one

def run_batch(items, call_model, attempts=2):
    for attempt in range(1, attempts + 1):
        raw = call_model(build_prompt(items))
        answers, verdict = validate(items, raw)
        if verdict == "ok":
            return answers, {"attempts": attempt, "verdict": verdict, "fallback": False}

    out = {}
    for item in items:
        raw = call_model(build_prompt([item]))
        answers, verdict = validate([item], raw)
        out[item.id] = answers[item.id] if verdict == "ok" else None
    return out, {"verdict": "fallback_single", "fallback": True}

python batch_harness.py --input items.jsonl --batch-size 2 --ledger runs/ledger.jsonl
tail -n 3 runs/ledger.jsonl

{"batch": ["9f2c41ab07de", "41ab9f2c07de"], "attempt": 1, "verdict": "set_mismatch", "extra": ["7d10c3a1f0b2"], "fallback": true, "model_string": "<record what you actually got>"}

The ledger excerpt above is illustrative — the values are placeholders, and the deliverable is the columns, not the numbers. Record the model string you actually received, because free access is not the same thing as a pinned version, and a drifting model is a variable you want in the log.

The run sheet I would keep

Column	Why it earns its place
`batch` (sorted ids)	Reproducing a failure needs the exact inputs, not the item count
`attempt`	Tells you whether retries are rare or structural
`verdict`	Distinguishes `not_json` from `set_mismatch`, which are different bugs
`extra` / `missing`	The invented ids are the signal that the prompt drifted
`fallback`	A rising fallback rate means the batch size is too aggressive

And a decision table for batch size, which is really a question about failure cost:

Items per call	Use it when	Do not use it when
1	Items are long, or ordering is contractual	You are already comfortable with the fallback path
2	Items are short, independent, and id-stable	One bad item can poison its neighbour
4+	You have eyes on the ledger during the run	You cannot inspect the first hundred rows by hand

What I would repeat, and what I would not

I would repeat the input-derived id and the strict set-equality check on day one, not day two; both are ten lines and they replace a whole class of late-night doubt. I would repeat keeping every model call behind one function, because it made the fallback trivial to write. I would not repeat starting at batch size two before watching a hundred single-item calls go through cleanly. The boring baseline is what tells you whether batching is safe for your particular items.

Who should not do this

Anyone whose items reference each other, since batching changes what the model can see at once.
Anyone under a hard latency or availability guarantee, because a free endpoint is a shared one and I make no promises about it.
Anyone who cannot hand-inspect the first hundred outputs, since the validator checks shape and nothing else.
Anyone paying per token on a long shared instruction, where duplicating it per item costs more than the round trips you save.

The harness does not know whether an answer is correct — it only knows that every id it sent came back once. That is a smaller promise than it sounds, and it is the one I can actually keep. If you run this drill on the free model access, start at batch size one and let the ledger tell you when two is safe.

48-Hour Field Notes: The Model Was Fast — My Connection Setup Was the Queue

Taylor Wang — Mon, 26 Oct 2026 21:19:16 +0000

Background: a worker I could not profile

I had a batch job that pushed a few hundred short prompts at a model endpoint and wrote every answer to disk as it arrived. It ran on a free server box, it used a free model tier, and it felt slow in the vaguest possible way. My first instinct was to blame the model, because "the model is slow" is a story I can tell without opening a profiler. That instinct survived almost two full days, and what finally killed it was a twenty-line timing script.

Ask yourself a blunt question: when your job is slow, which phase are you actually naming? If the answer is "the call", you have not named anything yet.

The measurement I should have run on hour zero

Every HTTPS request is at least five phases pretending to be one. Collapse them and the only part you control disappears.

dns — resolving the hostname, which may hit a resolver cache or miss it entirely.
tcp — the socket connect, which is where a shared box's CPU quota shows up first.
tls — the handshake, which costs a round trip or two before any HTTP exists.
ttfb — from request bytes sent to the first response byte, the closest thing you have to "model time".
body — reading the response, which is buffering behaviour more than compute.

Here is the harness I ended up keeping. It is stdlib only, so it runs on a free box without a dependency install.

#!/usr/bin/env python3
# phase_timer.py - split one HTTPS call into dns / tcp / tls / ttfb / body
import json, socket, ssl, sys, time
from urllib.parse import urlsplit


def now_ms():
    return time.perf_counter() * 1000.0


def timed_call(url, headers=None, timeout=10.0):
    u = urlsplit(url)
    host, port = u.hostname, u.port or 443
    path = u.path or "/"
    row = {"url": url}

    t = now_ms()
    infos = socket.getaddrinfo(host, port, type=socket.SOCK_STREAM)
    row["dns_ms"] = now_ms() - t

    family, socktype, proto, _, addr = infos[0]
    t = now_ms()
    sock = socket.socket(family, socktype, proto)
    sock.settimeout(timeout)
    sock.connect(addr)
    row["tcp_ms"] = now_ms() - t

    t = now_ms()
    ctx = ssl.create_default_context()
    tls = ctx.wrap_socket(sock, server_hostname=host)
    row["tls_ms"] = now_ms() - t

    req = f"GET {path} HTTP/1.1\r\nHost: {host}\r\nConnection: close\r\n\r\n"
    t = now_ms()
    tls.sendall(req.encode())
    first = tls.recv(1)
    row["ttfb_ms"] = now_ms() - t

    t = now_ms()
    chunks, total = [first], len(first)
    while True:
        b = tls.recv(65536)
        if not b:
            break
        chunks.append(b)
        total += len(b)
    row["body_ms"] = now_ms() - t
    row["bytes"] = total
    tls.close()
    return row


if __name__ == "__main__":
    url, runs, out = sys.argv[1], int(sys.argv[2]), sys.argv[3]
    with open(out, "a", encoding="utf-8") as fh:
        for _ in range(runs):
            fh.write(json.dumps(timed_call(url)) + "\n")
            fh.flush()

Run it cold first, one fresh connection per call, and keep the file:

python3 phase_timer.py "https://your-endpoint.example/v1/models" 200 phases_cold.jsonl

Then run it again with one connection reused per worker and diff the two files. The gap between phases_cold.jsonl and phases_warm.jsonl is the entire article, and it is reproducible on your own box in under an hour.

What broke, in order of embarrassment

1. I opened a new connection for every single prompt

Each call paid the full handshake tax before a single token of work happened. On a shared free box with a small CPU quota, that tax is not a constant — it moves with whatever else is running on the host. So my "model latency" was partly a measurement of my neighbours, which is exactly the kind of number that makes you tune the wrong knob for two days.

2. My retry wrapper rebuilt the socket and called itself resilient

The slowest calls in the run were not the first attempts, they were the retries, because each retry re-ran dns, tcp and tls before it re-sent anything. A retry that reconnects is correct when the failure was in the connection, and wasteful when the failure was a stall on a socket that was already healthy. Does your retry know the difference? Mine did not, and it was hiding in a decorator I had stopped reading.

3. My timings had totals but no phase labels

I could prove the run was long, and I could not prove which part was long, so every hypothesis sounded equally plausible. A wall-clock total is a feeling with a decimal point. The fix was not more logging, it was logging the right five numbers per attempt.

Phase-attributed retries, which is the part worth copying

Once each attempt carries a phase, the retry policy writes itself. Pre-request failures mean nothing was sent, so retrying costs you almost nothing. Post-first-byte failures mean the provider already did the work, so a blind re-send can duplicate it.

RETRYABLE = {"dns", "tcp", "tls", "ttfb"}
CONNECTION_PHASES = {"dns", "tcp", "tls"}


def should_retry(phase, attempt):
    if phase not in RETRYABLE:
        return False
    if phase in CONNECTION_PHASES:
        return attempt < 2          # reconnect, then give up quietly
    return attempt < 4              # provider-side stall: back off, then stop

Two consequences fell out of that table immediately. Connection failures stopped being reported as model failures, and my fallback path stopped firing on problems the fallback could not fix. If you only change one thing after reading this, make it that: tag the phase, then let policy depend on it.

A decision table I keep next to the harness

What the JSONL shows	Most likely cause	What I change
`tcp_ms` + `tls_ms` comparable to `ttfb_ms`	new socket per call	one connection per worker, keep-alive on
`dns_ms` spikes with no pattern	resolver misses on a restarting box	resolve once per process, reuse the address
`ttfb_ms` p90 much larger than p50, body flat	provider-side queueing	cap concurrency, do not add workers
`body_ms` grows with payload, ttfb flat	response buffering	read incrementally, time the first chunk
every phase flat but run long	your own code between calls	profile the loop, not the network

Where MonkeyCode actually fit

I used MonkeyCode's free model access to run the calibration passes, since a few hundred throwaway calls per iteration is exactly what you do not want to pay for while you are still guessing. The free server option hosted the harness next to the worker, so the numbers came from the same box that ran the job instead of from my laptop. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The free tier is also shared and noisy, which is precisely why the harness matters: do not trust my shapes, measure yours.

Limitations, and who should skip this

One box, one region, one endpoint. This workflow does not tell you anything about multi-region routing or provider-side rate limits.
Small samples are noise. Under roughly fifty calls, p90 is a rumour, so run for a full day before drawing a conclusion.
A harness that reuses a socket will not show you TLS session resumption, connection pooling internals, or HTTP/2 behaviour.
If you have a hard latency SLO and a paid provider with a status page, this is the wrong tool; go read their numbers instead of generating your own.
If your calls are long-lived streams where the first byte is the only byte that matters, skip the body phase and stop here.

What I would repeat at hour 48

Hours 0–2: run the cold harness, save the JSONL, and compute the share of each phase before forming any opinion.
Hours 2–6: flip to one connection per worker, re-run the identical command, and diff the two files.
Hours 6–24: run the real job with phase-tagged attempts, and log the phase on every failure, not just the message.
Hours 24–48: change one variable per run, then re-read the p90 instead of the average, because the tail is what misses deadlines.

If you run the harness yourself, keep the JSONL and diff it after each change. That file, not my conclusions, is the artifact worth comparing with someone else's box.

48-Hour Field Notes: My Overnight Batch Forecast Used the Mean and Ignored the Restart Tax

Taylor Wang — Mon, 26 Oct 2026 18:13:12 +0000

Have you ever sized an overnight batch job on paper, watched it finish comfortably inside the estimate, and then found it still churning at breakfast? I have, twice this month, and both times the arithmetic was correct and the assumption underneath it was wrong. This is a 48-hour field note about the difference between an average and a promise, and about the hidden cost of every restart on an interruptible box.

The setup was ordinary: a few thousand items, one model call per item, a checkpoint file so nothing gets done twice, and a machine that can reboot under me. I ran it on MonkeyCode's free model access and its free server option, which is what made the restart question unavoidable rather than theoretical. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Hour 0–12: the pilot I should have run first

My first version of the forecast was one line: item count times average call time. It looked responsible because I had measured something instead of guessing, and the measurement came from a sixty-item sample I had actually run. What I never asked was whether that sample resembled the tail of the distribution, or whether the box would stay up for the whole window.

Two things broke that forecast, and neither of them was the model being slow on average.

The distribution has a tail. Most calls returned quickly, a few took several times longer, and the queue inherited the tail because it processes every item once.
The checkpoint granularity set the restart tax. Saving every tenth item meant a reboot could replay up to nine finished calls, and I had counted that as zero.

Neither problem shows up in a mean. Both show up in a p90 and in a replay counter.

Hour 12–36: measuring instead of assuming

The harness below is the shape I now use before committing to any overnight run. Treat the printed numbers in the next section as illustrative output, not as a benchmark of any provider.

# pilot.py — run this on the same box that will run the batch
import random, statistics, time, json


def pct(values, p):
    s = sorted(values)
    if not s:
        return float("nan")
    k = min(len(s) - 1, max(0, round((p / 100) * (len(s) - 1))))
    return s[k]


def pilot(items, call, sample=60, seed=7):
    rng = random.Random(seed)
    picks = rng.sample(items, min(sample, len(items)))
    rows = []
    for i, item in enumerate(picks):
        t0 = time.monotonic()
        try:
            out = call(item)
            ok = out is not None
        except Exception as exc:          # record, do not swallow
            out, ok = repr(exc), False
        rows.append({
            "i": i,
            "seconds": round(time.monotonic() - t0, 3),
            "ok": ok,
        })
    return rows

Then the forecast itself, which is deliberately pessimistic in two places. It prices items at p90 rather than the mean, and it multiplies by a restart tax you measured rather than one you imagined.

# forecast.py — pessimistic on purpose
def forecast(n_items, rows, p=90, checkpoint_every=1, restarts=1):
    ok = [r["seconds"] for r in rows if r["ok"]]
    failures = len(rows) - len(ok)
    per_item = pct(ok, p)
    # worst case: each restart discards up to checkpoint_every-1 finished items
    replay = restarts * (checkpoint_every - 1)
    billed_items = n_items * (1 + failures / max(len(rows), 1)) + replay
    total_s = billed_items * per_item
    return {
        "p50_item_s": pct(ok, 50),
        "p90_item_s": pct(ok, p),
        "sample_failures": failures,
        "billed_items": round(billed_items, 1),
        "eta_hours": round(total_s / 3600, 2),
    }

If the free server can be restarted by something outside your control, then restarts is not a what-if parameter, it is a budget line. I set it from observed behaviour over the pilot window and then doubled the checkpoint interval's worst case on purpose.

Measuring the restart tax with a kill, not a guess

You cannot estimate replay cost by reading your checkpoint code, because the real number depends on when the interrupt lands. So I interrupt it on purpose, twice, at known item counts.

# terminal 1
python run_batch.py --limit 400 --checkpoint ./ckpt.jsonl

# terminal 2, after ~90 seconds
kill -TERM $(pgrep -f run_batch.py)
wc -l ./ckpt.jsonl            # finished items at the moment of death
python run_batch.py --resume ./ckpt.jsonl --limit 400

The difference between the checkpoint line count and the item the worker was holding is your replay window in practice. Divide replayed items by total items and you have a tax rate you can put into the forecast instead of a hope.

Hour 36–48: the decision table I now keep

Not every batch deserves this ceremony, and the table below is how I decide in under a minute whether the harness pays for itself.

Workload shape	Run the pilot?	Where it should run
Under ~200 items, finishes in minutes	No	Anywhere, including a laptop
Thousands of items, idempotent per item	Yes, 60-item sample	Interruptible box with a checkpoint file
Per-item side effects that must happen once	Yes, plus an idempotency key	Needs a durable queue, not a script
Latency-sensitive, user waiting	No	This is a different problem entirely
Deadline inside a single request	No	Batching is the wrong tool

Three checks before I trust any forecast: does the pilot run on the same machine class, does the sample include a cold start, and is the p90 computed from at least sixty successful items? If any answer is no, the forecast is a rumour with decimal places.

What I would repeat, and what I would not

I would repeat the deliberate kill. It cost two minutes and replaced a paragraph of optimism with a number, and it also proved the resume path actually resumes instead of silently restarting from zero. I would repeat pricing at p90 even though it made my estimate look worse, because the estimate exists to protect the morning, not the ego.

I would not repeat two things. First, sampling only when the box is idle, since a pilot measured under no load is a best case wearing a lab coat. Second, checkpointing every tenth item on a job where the expensive part is the model call rather than the disk write — flushing a small JSONL line per item is cheap, and the replay window shrinks by an order of magnitude.

Limitations, and who should skip this

A sixty-item pilot is a smoke test, not a statistics project, and it will mislead you if latency drifts by time of day or if your item mix is not uniform. Free infrastructure is shared by definition, so treat any p90 as a snapshot with a date on it rather than a guarantee. If your job has hard external deadlines, contractual latency, or per-item effects that must occur exactly once, this harness is not enough and you need durable orchestration instead.

Skip this entirely for short jobs, for one-off scripts you will never run again, and for anything where the cost of the pilot exceeds the cost of just running the batch and watching it.

If you want to try the harness cheaply, run the sixty-item pilot and the two forced restarts against a free model and a free server before you commit to the full run — the numbers you get back will tell you more about your own code than any estimate I could hand you.

48-Hour Field Notes: My Free Box Looked Slow Until I Logged the Gap Between Requests

Taylor Wang — Mon, 26 Oct 2026 15:07:53 +0000

Everyone says the same sentence when a cheap box misbehaves: "the first request after a quiet night takes forever." But is that one bug, or is it three different bugs wearing the same coat, and would you even be able to tell them apart with the logs you keep today? My first attempt at answering that produced a nice-looking chart and a completely wrong conclusion, because I never recorded how long the box had been idle before each request.

So these are field notes from a 48-hour probe I built around a single question: what does the first request actually cost, and how would I know? Everything below is the harness and the decision rules, not a benchmark.

Why "it felt slow" is not a diagnosis

A slow-but-successful response, a 502 from a proxy, and a connection that never opens are three different worlds. The first one means the work happened; the second means something in front of the work was not ready; the third means you never reached anything at all. Retrying all three the same way is how people end up blaming a rate limit for a retry storm they wrote themselves, which is a lesson I already paid for once.

Ask yourself what your current logs would tell you if I asked: how many seconds had passed since the previous request? If the answer is "nothing", your latency numbers are mixing two populations of requests and averaging them into noise.

The 48-hour plan

Pick the one route your job actually calls first — not /health, not a TCP ping.
Probe it on a fixed interval, and record the gap since the previous probe in every record.
Never retry inside the probe. One attempt, one line, done — the probe measures, the job decides.
Write append-only JSONL so a crash costs you one line rather than the file.

Step 2 is the whole trick, and it is the step I skipped on the first run.

The prober

# probe_idle.py — one attempt per invocation, driven by cron or a systemd timer
import json, os, time, datetime, pathlib, urllib.request, urllib.error

ENDPOINT = os.environ["PROBE_URL"]   # the same first route your job hits
LOG = pathlib.Path(os.environ.get("PROBE_LOG", "probe.jsonl"))


def now_iso():
    return datetime.datetime.now(datetime.timezone.utc).isoformat()


def last_ts():
    if not LOG.exists():
        return None
    with LOG.open("rb") as f:
        f.seek(0, os.SEEK_END)
        f.seek(max(0, f.tell() - 4096))
        tail = f.read().decode("utf-8", "ignore").strip().splitlines()
    return json.loads(tail[-1])["ts"] if tail else None


def probe():
    t0 = time.perf_counter()
    status, err = None, None
    try:
        req = urllib.request.Request(ENDPOINT, headers={"User-Agent": "idle-probe/1"})
        with urllib.request.urlopen(req, timeout=20) as r:
            status = r.status
            r.read(1)                       # read a byte so "headers only" is not the measurement
    except urllib.error.HTTPError as e:
        status = e.code
    except Exception as e:
        err = type(e).__name__
    return status, err, round((time.perf_counter() - t0) * 1000)


if __name__ == "__main__":
    ts, prev = now_iso(), last_ts()
    status, err, ms = probe()
    gap = None
    if prev:
        gap = round((datetime.datetime.fromisoformat(ts)
                     - datetime.datetime.fromisoformat(prev)).total_seconds())
    rec = {"ts": ts, "gap_s": gap, "ms": ms, "status": status, "err": err}
    with LOG.open("a") as f:
        f.write(json.dumps(rec) + "\n")
    print(rec)

Run it every 5 minutes for the first day, then every 30 minutes, and compare the two days. Yes, changing the interval is a deliberate perturbation of the system you are observing — hold that thought, it becomes a problem in a minute.

Illustrative output (synthetic, not measured)

The lines below are made-up examples that show the shape of the data; substitute your own run before drawing any conclusion:

{"ts": "2026-10-25T02:00:04+00:00", "gap_s": null,  "ms": 1840, "status": 200, "err": null}
{"ts": "2026-10-25T02:05:05+00:00", "gap_s": 301,   "ms": 190,  "status": 200, "err": null}
{"ts": "2026-10-25T09:30:11+00:00", "gap_s": 15906, "ms": 7310, "status": 200, "err": null}
{"ts": "2026-10-25T09:35:12+00:00", "gap_s": 301,   "ms": 205,  "status": 200, "err": null}
{"ts": "2026-10-25T14:00:03+00:00", "gap_s": 15891, "ms": 20,   "status": 502, "err": null}

Classifying before you conclude

Latency alone cannot separate the three failure kinds, so classify each record first and only then summarise.

import json, statistics as st

FAST_MS = 1500

def cls(r):
    if r["err"]:
        return "conn_error"          # timeout, refused, DNS
    if r["status"] >= 500:
        return "http_5xx"            # reached something, it was not ready
    if r["status"] >= 400:
        return "http_4xx"            # your request is the problem, not the box
    return "ok" if r["ms"] < FAST_MS else "ok_slow"

rows = [json.loads(l) for l in open("probe.jsonl")]
for r in rows:
    r["cls"] = cls(r)

cold = [r for r in rows if r["gap_s"] and r["gap_s"] > 900]
warm = [r for r in rows if r["gap_s"] and r["gap_s"] <= 300]

for name, bucket in (("cold", cold), ("warm", warm)):
    ok = [r["ms"] for r in bucket if r["cls"].startswith("ok")]
    print(name, "n=", len(bucket),
          "ok=", len(ok),
          "p50=", st.median(ok) if ok else None,
          "fail=", sum(1 for r in bucket if not r["cls"].startswith("ok")))

If cold and warm look like different systems, congratulations — you have two populations, and your monthly average was hiding both of them.

The decision table I actually wanted

Observed class	What it usually means	Probe action	Job action
`ok_slow` on first request, fast after	work is being done lazily on the cold path	record it, do not retry	warm once, then run
`http_5xx` on first request	something in front of the app is not ready	record it, do not retry	back off, then one warm-up, then run
`conn_error` (timeout/refused)	nothing accepted the connection	record the gap	bounded retry with jitter, then fail loudly
`4xx`	your own request shape	stop probing	fix the client

Notice the probe never retries in any row. The probe's job is to make the world legible; the job's job is to survive it.

What broke, and what I would repeat

The keepalive confound. Probing every five minutes may keep the box warm, which means you can no longer observe the thing you set out to measure. I would run a second, sparse phase at 30-minute intervals and compare, rather than trusting a single cadence.
A missing covariate beat me on day one. Without gap_s, a 7-second response and a 7-second response after nine hours of silence are indistinguishable rows — one number, two stories.
/health lied by being easy. A static health route can be warm while the real first route is still doing lazy setup; probe the route your job calls, not the one that is cheapest.
Warm-up before the job, not around the clock. A 24/7 keepalive buys you a permanently busy box and hides the cold-path failure your users will still hit at 3 a.m.

I would repeat the append-only JSONL, the no-retry probe, and the cold/warm split. I would not repeat the single-cadence run, and I would add one column for the response's Content-Length so a truncated body shows up as a number instead of a mystery.

If your first request after idle is a model call, the cold path includes the provider too, so the probe has to hit a model-backed route to be honest about it. That is the one place MonkeyCode's free model access and free server option fit here — they are operator-supplied availability claims, useful for running a 48-hour probe without a paid box while you are still deciding, not measured facts in this article. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Who should not use this

Skip this if you have real server-side latency histograms already — your tracing backend answers the cold/warm question better than a cron job does. Skip it if your traffic is steady enough that idle gaps barely exist, and skip it if a wrong guess costs nothing, because 48 hours of probing is not free either. And do not turn a two-day sample into a promise about permanence; a probe measures your box, on your days, against your route.

What I want from you is not agreement but a number: how many seconds passed since the previous request, and does your log know?

If you want to run the two-phase version before spending anything, the free server option is the cheapest place to park the timer — bring your own probe.jsonl and let the gap column argue with you.

DEV Community: Taylor Wang

Two Duplicate Rows in 48 Hours: My Unique Index Treats NULL as a New Value

What the job was supposed to guarantee

The two rows that should not exist

Why the database happily said yes

A reproduction you can run in five minutes

Step 1 — the schema I actually shipped

Step 2 — the test that proves the index does nothing

Step 3 — confirm the same insert is blocked when the hash is present

Fixes, ranked by how much they change

Clean up before you tighten the index

The ON CONFLICT trap I hit on attempt two

Who should not take this route

What I would repeat in the next 48 hours

48-Hour Field Notes: My Latency Numbers Were Wrong Because I Started the Clock in the Wrong Place

The three numbers I actually needed

The mistake: one time.time() for four different intervals

Rebuild: one row per attempt, four timestamps

The three queries that make the mistake obvious

Read the shape, not the score

Where MonkeyCode's free model access and free server fit in this workflow

What broke, and what I would repeat

Who should not use this approach

The 20-minute test plan I now run first

The question I should have asked on day one

48-Hour Field Notes: My 30-Second Timeout Didn't Cancel Anything, and the Server Logged Three Requests

What I Actually Believed

The Harness: A Server That Ignores You

Reading the Log Was the Whole Lesson

The question I now ask first

The Fix: An Intent Ledger, Not a Retry Counter

Decision Table I Now Keep Next to the Code

Where the Free Tier Actually Helped, and Where It Didn't

Limitations worth stating plainly

Who should not use this approach

What I'd Repeat in the Next 48 Hours

48-Hour Field Notes: My Homemade Eval Set Was Scoring My Prompt, Not the Model

What I thought I was measuring

The bug: my prompt examples were inside my eval

Artifact 1: a shingle overlap gate for prompt/eval leakage

The second bug: one sample per item is a coin flip

Where free access and a free server actually fit

What broke, and what I would repeat

A comparison table that survives price being zero

Limitations, and who should skip this

48-Hour Field Notes: I Ramped Worker Threads From 1 to 12 and Watched Throughput Fall Off a Cliff at 8

What I was actually optimizing

The product side, briefly

Attempt 1: more threads, less thinking

The harness I should have written on day one

What one sweep showed

What broke, and what the failure was not

Decision table

Who should not use this approach

What I would repeat, and what I wasted time on

48-Hour Field Notes: I Crash-Tested My Resumable Batch Job Before Trusting a Free Box

The setup, and the constraint that shaped it

The resume contract: three states, one fenced write

Crash injection: making failure reproducible

What broke in the first 24 hours

When the free box is the right place for this

Limitations I will not pretend away

What I would repeat

48-Hour Field Notes: My Queue Drained to Zero and the Work Wasn't Done

Hour 0-6: the counter that lied

Hour 6-18: two writes, no transaction

The claim needs a lease, not a flag

Hour 18-30: the restart duplicated calls

Hour 30-42: the key needed the prompt version

Hour 42-48: the kill test I would repeat

A decision table for the free path

Who should not use this approach

What I would repeat on Monday

48-Hour Field Notes: My Prompt Budget Was in Characters, and the Overflow Never Raised

The budget I thought I had

Hour 0-6: build the fixture corpus before the pipeline

Hour 6-20: what actually broke

Hour 20-34: fail loudly, then truncate on purpose

A decision table I actually use

Hour 34-48: what I would repeat and what I would skip

The `ON CONFLICT` trap I hit on attempt two

The mistake: one `time.time()` for four different intervals