48-Hour Field Notes: I Stopped Asking "Did It Finish?" and Started Hashing Inputs

Saturday, 09:10. A batch job that enriches a few thousand records was supposed to take one afternoon. On Sunday it had "finished" three times, and I still could not tell you which rows were real, which were half-written, and which had been processed twice by a worker I never saw start.

Here is the part that annoyed me: nothing had crashed loudly. The status column said done, the log said done, and the output directory disagreed with both. So I spent the second day deleting my status field and replacing it with something that cannot lie.

What I Tried, In Order

I want to be honest about the sequence, because the first two attempts were wasted hours and the third one is the only thing I kept.

1. A status column in Postgres. Flipped to done after the write, so a kill between write and update left rows that existed but were invisible to the next run. Classic, and I knew better.
2. A lock file per item. Better, until the process died holding the lock, and now the item was permanently "in progress" with nobody working on it.
3. A content-addressed manifest. Each finished item gets a small JSON record containing its input hash and its output hash. Resume logic stops trusting any flag and re-derives the truth from bytes on disk.

The third one survived contact with reality. It also made my verification step trivial, which turned out to matter more than the resume logic itself.

The Artifact: A Manifest That Cannot Say "Done" Without Proof

This is the whole trick, and it fits in one file. No framework, no dependency beyond the standard library.

# manifest.py - content-addressed resume for a batch worker
import hashlib, json, os, pathlib, tempfile

ROOT = pathlib.Path(os.environ.get("JOB_ROOT", "./job"))
OUT, MAN = ROOT / "out", ROOT / "manifest"
SCHEMA = 2

def h16(data: bytes) -> str:
    return hashlib.sha256(data).hexdigest()[:16]

def fingerprint(item: dict) -> str:
    blob = json.dumps(item, sort_keys=True, separators=(",", ":")).encode()
    return h16(blob)

def write_atomic(path: pathlib.Path, data: bytes) -> None:
    path.parent.mkdir(parents=True, exist_ok=True)
    fd, tmp = tempfile.mkstemp(dir=path.parent)
    with os.fdopen(fd, "wb") as f:
        f.write(data); f.flush(); os.fsync(f.fileno())
    os.replace(tmp, path)

def done(item: dict) -> bool:
    rec_path, out_path = MAN / f"{item['id']}.json", OUT / f"{item['id']}.json"
    if not rec_path.exists() or not out_path.exists():
        return False
    try:
        rec = json.loads(rec_path.read_text())
    except json.JSONDecodeError:
        return False                     # truncated manifest is not "done"
    if rec.get("schema") != SCHEMA:
        return False                     # output format changed, redo it
    if rec.get("input_hash") != fingerprint(item):
        return False                     # source record changed since completion
    return rec.get("output_hash") == h16(out_path.read_bytes())

def commit(item: dict, result: dict) -> None:
    out_path = OUT / f"{item['id']}.json"
    write_atomic(out_path, json.dumps(result, sort_keys=True).encode())
    rec = {"schema": SCHEMA, "item_id": item["id"],
           "input_hash": fingerprint(item),
           "output_hash": h16(out_path.read_bytes())}
    write_atomic(MAN / f"{item['id']}.json", json.dumps(rec).encode())

Two details carry all the weight. write_atomic fsyncs into a temp file and then renames, so a kill either leaves the old state or the new one, never a truncated JSON body. And done() re-hashes the output on every call, which costs milliseconds per item and buys me the ability to point at any row and prove it.

The Verifier Is the Point, Not the Resume

Resume saves money; verification saves your weekend. The same module answers "what is actually true right now" in two sets:

import json

def verify(items: list[dict]) -> dict:
    ids = {i["id"] for i in items}
    return {"missing": sorted(i for i in ids if not done(next(x for x in items if x["id"] == i))),
            "orphans": sorted({p.stem for p in OUT.glob("*.json")} - ids)}

missing means work the manifest refuses to vouch for. orphans means output on disk with no matching input, which is usually the fingerprint of a previous run against a different dataset. Both are one-line checks you can drop into CI.

Where the Free Tier Fit, and What I Refused to Assume

My worker ran on MonkeyCode's free server option, and the extraction step called free model access through the same account, so the whole 48 hours cost me nothing but time. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I verified my own pipeline behaviour, not the platform's internals, so treat the plan below as a harness you run against your workload rather than a review of anyone's infrastructure.

The uncomfortable part of any free tier is that you do not control restarts, and you should not design as if you do. I assumed nothing about permanence, nothing about notice before a restart, and nothing about how long a single job may run. Those are unknowns I cannot resolve from the outside, so the harness measures them instead of guessing.

A Pre-Flight Contract You Can Run in Ten Minutes

Assumption I was making	How I tested it	What failure looks like
Long jobs survive	Append a timestamped heartbeat file every 30s, watch it	Heartbeat stops, process gone
Restart means resume, not duplicate	Run the same 5 items twice, count manifest entries	Count doubles
Writes are durable across restart	kill -9 mid-write, then verify()	missing non-empty, orphans grows
Model output parses	Strict json.loads on 20 samples, log rejects	Reject rate above zero
The box is reachable when I sleep	Cron job pinging a health endpoint at 03:00	Silence

Run that table before the long job, not after. The kill -9 row is the one people skip, and it is the only row that proves your resume logic instead of your optimism.

What Broke, Precisely

• The manifest lived on the same volume as the output. A volume reset took both, and my verifier happily reported zero missing items because there was nothing left to compare. Manifest and output now live in separate paths so a partial reset is loud.
• Model output drifted in shape. One record came back with the field nested one level deeper, my commit wrote it, and the hash was perfectly valid — for a wrong document. A schema version in the manifest record forces a redo when the contract changes.
• I retried inside the worker and outside it. Two retry layers multiplied instead of adding, which is a mistake I have now made twice in my life and would like to stop making.

Who Should Not Use This

If your job has ten items and runs in a minute, done() is ceremony — just re-run everything. If your outputs are large binaries, hashing them on every check gets expensive and you want a cheaper sentinel plus a periodic sweep. And if you genuinely need guaranteed uptime, a free server option is the wrong dependency regardless of how good your resume logic is; the manifest makes interruption survivable, not impossible.

What I Would Repeat

Assume the process will die between the write and the bookkeeping, then design the bookkeeping so it can be recomputed from the write. Keep the verifier in the same file as the resume logic, because the moment they live apart they start disagreeing. And run the kill test on day one instead of day two.

If you adopt the manifest harness, wire verify() into your post-run check before you trust any green status field — that single call is what turned my Sunday from archaeology into a two-minute glance.