48-Hour Field Notes: My Batch Job Trusted Local Disk, and a Restart Re-Bought Every Token

I had a batch job that walked a queue of documents, called a model for each one, and wrote a summary into an output table. It ran fine for six hours on a small worker, and I felt clever about the whole arrangement. Then the worker restarted, the queue replayed from the top, and every item got processed a second time. Did I lose data? No. I lost something I had not been measuring at all: repeated work, repeated latency, and a warm fuzzy feeling that turned out to be unfounded.

This is the write-up of the next 42 hours: what I tried, what broke, and the small receipts table I now build before I build anything else.

The setup, and the assumption I never wrote down

I was running a summarization batch on MonkeyCode's free server option and calling models through its free model access, because I wanted to spend my effort on the pipeline instead of on a bill. So far so good. The problem was not the platform and not the model; the problem was that I had quietly assumed three things and verified none of them.

• That local disk on the worker survives a restart long enough for my checkpoint file to matter.
• That "the job already ran" is something the pipeline can figure out from its own state.
• That a restart is roughly the same event as a crash, so the same recovery logic covers both.

None of those are platform facts, and none of them are model facts. They are assumptions about my code, and I had written zero lines to test them. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Hour 0–6: the smoke test that lied by being too small

My smoke test processed five items serially, printed five lines of output, and exited cleanly. Excellent. But a serial run has no lease, no contention, and nothing to resume, so it exercises roughly none of the failure modes that matter at batch scale. Fair question: why did I trust it? Because it was fast and green, and green output is a terrible proxy for correct recovery behaviour.

The run itself was simple: claim an item, build a prompt, call the model, write the result, mark the item finished. Notice the ordering. The "mark finished" step happens after the call returns, which means there is a window where the model has done the work and my process has not recorded it. That window is fine if it is rare and cheap. It is not fine if recovery replays the entire queue.

Hour 6–18: the restart, and the replay

The worker restarted mid-run, and the resume path found no checkpoint it recognised. So it did what I told it to do: it started over. Every completed item got re-sent, and because nothing downstream was keyed on content, every duplicate looked like brand-new work.

Here is what I would tell anyone in that moment, and what I had to tell myself slowly: a resume mechanism that depends on the worker's own filesystem is a resume mechanism that only works while the worker is the same worker. If you cannot state, in one sentence, where the durable truth lives, you do not have one.

Numbers in this post are illustrative, because the useful part is the shape of the failure rather than my exact volume. The shape was: work in, restart, work in again, no record that the work had ever happened.

Hour 18–30: a receipts table instead of a checkpoint file

I replaced the checkpoint file with a table whose primary key is derived from the content of the work rather than from a timestamp, a sequence number, or a worker id. Every call has a key, every key has a row, and the row is written before the call goes out.

import hashlib, json, sqlite3, time

SCHEMA = """
PRAGMA journal_mode=WAL;
CREATE TABLE IF NOT EXISTS receipts (
  idem_key    TEXT PRIMARY KEY,
  item_id     TEXT NOT NULL,
  stage       TEXT NOT NULL,
  state       TEXT NOT NULL CHECK (state IN ('claimed','done','failed')),
  owner       TEXT NOT NULL,
  claimed_at  INTEGER NOT NULL,
  finished_at INTEGER,
  output_ref  TEXT
);
CREATE INDEX IF NOT EXISTS receipts_state_idx ON receipts(state, claimed_at);
"""

def idem_key(item_id, stage, template_version, model_alias):
    payload = {"item": item_id, "stage": stage,
               "tpl": template_version, "model": model_alias}
    blob = json.dumps(payload, sort_keys=True, separators=(",", ":")).encode()
    return hashlib.sha256(blob).hexdigest()

def connect(path):
    conn = sqlite3.connect(path, isolation_level=None, timeout=10)
    conn.executescript(SCHEMA)
    conn.execute("PRAGMA busy_timeout = 10000")
    return conn

def claim(conn, key, item_id, stage, owner, lease_s=900):
    now = int(time.time())
    conn.execute("BEGIN IMMEDIATE")
    try:
        row = conn.execute(
            "SELECT state, claimed_at FROM receipts WHERE idem_key = ?", (key,)
        ).fetchone()
        if row is None:
            conn.execute(
                "INSERT INTO receipts (idem_key, item_id, stage, state, owner, claimed_at)"
                " VALUES (?, ?, ?, 'claimed', ?, ?)",
                (key, item_id, stage, owner, now))
            conn.execute("COMMIT")
            return True
        state, claimed_at = row
        if state == "done" or (state == "claimed" and now - claimed_at < lease_s):
            conn.execute("COMMIT")
            return False
        conn.execute(
            "UPDATE receipts SET state='claimed', owner=?, claimed_at=?"
            " WHERE idem_key=? AND state IN ('claimed','failed')",
            (owner, now, key))
        conn.execute("COMMIT")
        return True
    except Exception:
        conn.execute("ROLLBACK")
        raise

def complete(conn, key, owner, output_ref):
    conn.execute(
        "UPDATE receipts SET state='done', finished_at=?, output_ref=?"
        " WHERE idem_key=? AND owner=?",
        (int(time.time()), output_ref, key, owner))

Three details earn their keep. BEGIN IMMEDIATE takes the write lock up front instead of upgrading mid-transaction. The owner=? guard means a worker whose lease was reassigned cannot quietly mark the row done. And output_ref points at the artifact instead of storing it, so the receipts table stays small enough to query while the job runs.

Hour 30–42: the probe I should have run first, and the queries I now keep

Before the next run, I wrote a five-minute durability probe. It is not clever, and that is the point: it produces a fact where I previously had a vibe.

mkdir -p state && date -u +%FT%TZ > state/boot.marker
hostname >> state/boot.marker
# now restart the server the way your plan actually restarts it
cat state/boot.marker || echo "marker gone: treat local disk as ephemeral"
hostname

Run it twice and write down four results: whether the marker file survived, whether the hostname changed, whether the working directory contents are the same, and whether the clock jumped. Whatever you did not test, treat as not surviving. That single rule removed an entire class of surprises for me.

Two queries also became part of the run report, because I do not trust a pipeline I cannot interrogate:

-- did anything finish twice? usually means the key inputs drifted
SELECT item_id, stage, COUNT(*) AS done_rows
FROM receipts WHERE state = 'done'
GROUP BY item_id, stage HAVING COUNT(*) > 1;

-- how much work is currently in flight, and is any of it stale?
SELECT state, COUNT(*) FROM receipts
WHERE state = 'claimed' AND claimed_at < strftime('%s','now') - 900
GROUP BY state;

The first query is my favourite kind of alarm: it fires when I bump a template version or switch model aliases without thinking about the key, which is exactly the moment a duplicate would otherwise slip past silently.

What I would repeat, and what I would not

• Repeat: writing the durability probe before the pipeline, because it is cheaper than the replay.
• Repeat: keying on content plus template version plus model alias, so an intentional change produces new work and an accidental retry does not.
• Repeat: logging the idempotency key on every call, next to the latency and the output reference.
• Not repeat: storing the only copy of progress on the worker that produces the progress.
• Not repeat: letting a wall-clock value or a random UUID into the key material.
• Not repeat: treating lease expiry as an exception instead of a normal, expected event.

Is a free server the right home for your batch job?

Workload shape	Reasonable fit?	Guardrail you need first
One-shot enrichment, duplicates only cost time	Yes	Content-derived idempotency keys
Writes to external side effects (mail, tickets, payments)	Only with care	Pass the key to the downstream API's own dedupe
Single job outliving one lease	Split it	Stage boundaries with per-stage receipts
Depends on local files between steps	No, unless probed	External store, or verified persistence
Latency-sensitive request path	No	Use a warm, dedicated runtime
Data you cannot freely relocate	Depends on your policy	Decide before the first call, not after

Who should not follow this workflow

If you need a p99 latency guarantee, do not build on a worker whose lifecycle you do not control. If your work unit is one long stateful session, the claim-and-complete pattern will fight you, and you should shard by stage instead. If you cannot yet name where your durable state lives, stop designing the pipeline and answer that question first, because everything else is decoration.

Closing note

Forty-eight hours bought me a table with four columns and a probe script with three lines, and both of them are now the first thing I write for any batch job. The restart was not the failure; the failure was that I had never asked what a restart would do to my assumptions.

If you are wiring something similar and want to compare notes on receipt tables or restart probes, the free model access and free server option in MonkeyCode are a low-stakes place to run the experiment — bring your own durable store and your own probe.