48-Hour Field Notes: My Batch Job Survived the Restart but Came Back With a Checkpoint I Couldn't Trust

A process that comes back up is not the same thing as a job that resumes, and it took me two nights of staring at logs to accept that difference. My batch job was supposed to summarize a pile of text records and land one file per record; the wrapper restarted it cleanly every time, printed a friendly banner, and then quietly did almost nothing. So which part was lying to me — the restart, the checkpoint, or the definition of "done" I had written down?

These are field notes from a 48-hour window: what I tried, what broke on the second restart, and the small recovery contract I would repeat tomorrow. All the code below is runnable Python with no dependencies beyond the standard library, and the drill at the end lets you reproduce the failure in about five minutes on your own machine.

The Setup: Free Server, Free Model Access, One Naive Loop

I ran the job on MonkeyCode's free server option and sent each record to a model through its free model access, because I wanted to learn what a cheap nightly pipeline actually feels like before spending money on one. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Before you copy anything here, check the current docs yourself for quotas, storage behavior, and any runtime limits, since those change and a blog post is a terrible source of truth for them.

My original loop looked reasonable and was wrong in one specific way:

• Read every record into memory at startup, keyed by an id I computed from the source.
• Call the model, write the output file, then mark the record as done in a local SQLite file.
• On restart, skip anything already marked done and continue from there.

Can you spot the assumption? It is that the SQLite file and the output directory live on the same disk, survive the same events, and are updated in an order that leaves no gap. On an ephemeral host, at least one of those is usually false.

What Broke: Orphaned running Rows and a Disk That Forgot

First restart, nothing dramatic: the process died mid-batch, came back, skipped the finished records, and carried on. I felt clever. Second restart, the job printed the resume banner and then exited immediately with "0 records to process" while the output directory was visibly missing files.

The reason was a row state I had never thought about. When the process is killed between "mark running" and "mark done", that record stays running forever, and my resume query only looked for pending. The job had not resumed; it had confidently declared victory over the exact records it lost. Meanwhile a redeploy had given me a fresh filesystem, so the SQLite file remembered work that the output directory had never heard of.

That combination produced the worst kind of failure: a green exit code and a hole in the data. Since then I treat "done" as a claim that needs evidence, not a boolean I set optimistically.

The Fix: A Small Recovery Contract

The recovery contract is four rules, and none of them are clever. I wrote them down as checks so that a future me cannot quietly skip one of them.

1. Probe the state location before the first write, and print what you found, because a silent fallback to /tmp is how this whole genre of bug starts.
2. Claim work with a single atomic statement, so two processes or two restarts cannot both believe they own the same record.
3. Write output atomically, then record its hash, so a half-written file can never be mistaken for a finished one.
4. Recover orphans on startup, because a running row from a killed process is not progress, it is debris.

That is the whole idea. The hash is what makes rule three useful: if the row and the file disagree, the row loses.

Step 1: Probe the disk before you trust it

A single st_dev comparison will not prove durability, but it will catch the common case where my "persistent" path is actually on the same throwaway device as /tmp. That check has saved me twice.

import os
from pathlib import Path

def probe_state_dir(state_dir: Path, tmp_dir: Path = Path("/tmp")) -> None:
    """Fail loudly if the state dir looks ephemeral or is not writable."""
    state_dir.mkdir(parents=True, exist_ok=True)
    probe = state_dir / ".probe"
    probe.write_text("ok")
    if probe.read_text() != "ok":
        raise RuntimeError(f"state dir {state_dir} is not reliably writable")
    same_device = state_dir.stat().st_dev == tmp_dir.stat().st_dev
    print(f"[probe] state_dir={state_dir} same_device_as_tmp={same_device}")
    if same_device:
        print("[probe] WARNING: state may vanish on restart; point JOB_STATE_DIR at durable storage")

Should this be a hard failure? On a free server I prefer a loud warning plus an explicit --allow-ephemeral flag, because sometimes I genuinely want a throwaway run and I want to say so out loud.

Step 2: Claim work with one atomic statement

The state machine is pending -> running -> done, with failed available for records that exhausted their attempts. The claim is a conditional UPDATE, so the database decides who owns the record instead of my loop.

import sqlite3

def init_db(db_path: Path) -> sqlite3.Connection:
    conn = sqlite3.connect(db_path, isolation_level=None)
    conn.execute("PRAGMA journal_mode=WAL")
    conn.execute("""
        CREATE TABLE IF NOT EXISTS jobs (
            item_id    TEXT PRIMARY KEY,
            status     TEXT NOT NULL DEFAULT 'pending',
            attempts   INTEGER NOT NULL DEFAULT 0,
            output_sha TEXT
        )
    """)
    return conn

def claim(conn: sqlite3.Connection, item_id: str) -> bool:
    cur = conn.execute(
        "UPDATE jobs SET status='running', attempts=attempts+1 "
        "WHERE item_id=? AND status IN ('pending','failed')",
        (item_id,),
    )
    return cur.rowcount == 1

Notice what is missing: no read-then-write dance, no in-memory set of "already handled" ids. The database row is the only source of truth about ownership, and the file on disk is the only source of truth about output.

Step 3: Write output atomically, then record the hash

A partially written JSON file is indistinguishable from a complete one if you only check that the path exists. Writing to a temporary name and renaming is the cheap fix, and hashing on the way out gives the later verification pass something to compare against.

import hashlib, uuid

def write_atomic(path: Path, data: bytes) -> str:
    tmp = path.with_name(f"{path.name}.tmp.{uuid.uuid4().hex}")
    with open(tmp, "wb") as f:
        f.write(data)
        f.flush()
        os.fsync(f.fileno())
    os.replace(tmp, path)          # atomic within one filesystem
    return hashlib.sha256(data).hexdigest()

The os.replace guarantee is documented in the Python standard library reference, and it is the reason this pattern is boring and reliable instead of clever and fragile. The rename must happen on the same filesystem, so keep the temporary file next to the destination and never in /tmp.

Step 4: Recover orphans on startup

This is the step I originally skipped, and it is the one that turned a green exit code into missing data. Startup is not a fresh start; startup is a recovery pass that happens to also do new work.

def recover_orphans(conn: sqlite3.Connection) -> int:
    cur = conn.execute(
        "UPDATE jobs SET status='failed' "
        "WHERE status='running' AND attempts < 3"
    )
    cur2 = conn.execute(
        "UPDATE jobs SET status='pending' WHERE status='running'"
    )
    return cur.rowcount + cur2.rowcount

Running this before the main loop means a killed process no longer leaves records stranded forever. The two statements are ordered on purpose: anything with retries left goes back to pending, and anything that has burnt its attempts stays failed so it shows up in my report instead of looping quietly.

The Restart Drill You Can Actually Run

The drill matters more than the code, because the failure only appears when the process dies at an inconvenient moment. I now run this on every new batch job before it touches real data, and I would rather spend five minutes here than a night reading output files.

for attempt in 1 2 3; do
  python job_runner.py --limit 50 &
  pid=$!
  sleep 4
  kill -9 $pid        # no cleanup handlers, no graceful shutdown
  python job_runner.py --startup-check
done
python job_runner.py --verify

The --verify mode is three assertions, and any one of them failing means the checkpoint is lying to me:

• No row is stuck in running, because every orphan was either retried or parked as failed.
• Every done row has an output file whose SHA-256 matches the stored output_sha.
• The number of done rows matches the number of output files on disk, with no extras.

If you want a sharper test, kill -9 during the write rather than between writes. It is unpleasant, and it is exactly the scenario that a graceful SIGTERM handler will never teach you about.

Decision Table: Is This Worth the Extra Code?

Situation	Checkpoint + recovery pass?	Why
Minutes-long script, rerun is free	No	Rerunning is cheaper than maintaining state
Hours-long batch calling a paid or rate-limited API	Yes	Repeat calls cost real time and real quota
Writes to an external system with side effects	Yes, plus a boundary check	Your local done flag cannot undo a remote effect
Tiny dataset that fits in one request	No	There is nothing to resume
Long-lived service with a real queue	Use a durable workflow engine	You are rebuilding one badly

My rule of thumb: if a full rerun costs more than a coffee break, checkpoint it. If it costs less, delete the state and start clean, because a stale checkpoint is a worse problem than a wasted run.

Limitations and Who Should Not Bother

This approach assumes a single writer and a local or POSIX-like filesystem, so SQLite locking over a network share is a trap I would not walk into again. It also does not make model output deterministic, which means a retried record can produce a different summary than the one that vanished, and the hash only tells you the file matches the row, not that the content is stable.

On a free server option, confirm what storage actually survives a restart before you design around it, because I have assumed durability that was never there. If you need exactly-once effects in another system, or you are running more than one worker per shard, reach for a proper workflow engine instead of extending this pattern.

And if your job only takes ninety seconds, skip all of it. The correct amount of checkpointing for a fast idempotent script is zero, and I say that as someone who has proudly over-engineered plenty of them.

What I Would Repeat From These 48 Hours

I would repeat the startup recovery pass first, because it is ten lines that convert a silent hole into a visible report. I would repeat the kill -9 drill before the first real run, since graceful shutdown tests flatter a checkpoint design that has never met a crash. I would also repeat the habit of printing the state directory and its device id at startup, which sounds trivial until it explains an entire missing night.

What I would not repeat is trusting a green exit code as evidence of correctness. The job told me it finished, and it had finished exactly the records it could still remember.

If you end up trying the drill on your own free server setup, I would genuinely like to know which assertion failed first — mine was the orphan count, twice.