The Overnight Shift Workshop: Batch Jobs That Survive a Crash

A reading group met on a Tuesday evening to look at a nightly job that labeled support tickets with a model. The job had run for three weeks without complaint, then a container restart landed mid-batch and the next morning's report showed forty-one tickets tagged twice, eleven not tagged at all, and one row where two different labels sat in the same column. Nobody had written a bug. The job simply had no memory of what it had already done, and neither did anyone watching it.

That evening turned into the workshop below. It is ninety minutes long, it needs one laptop and one model endpoint, and its whole point is that the process will die without warning. Students do not debug a broken script. They build a job that can be killed on purpose and resumed by retyping the same command.

Why a free tier forces this lesson

The account's operator runs the drill on MonkeyCode's free model access and its free server option, and the program currently advertises 10 million free tokens. Treat those numbers as program terms rather than guarantees; the terms page, not a blog post, is the source of truth for what is left on any given day. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The free server matters more than the tokens here. Cheap compute is usually preemptible in practice, so the honest design assumption is that the worker disappears at an arbitrary second. A script that only works when nothing interrupts it is not a script, it is a demo. The workshop makes that difference visible in twenty minutes of wall-clock time.

The 90-minute outline

The first ten minutes are framing. The instructor puts the duplicate-ticket story on the board and asks one question: what would the job have to write down so that a restart changed nothing? Students shout "the id", "a timestamp", "whether it finished" — and the board slowly fills with the shape of a ledger.

Minutes ten through forty are the build. Each student writes a SQLite table with a stable job key, a hash of the input, a state column, and an attempt counter. They then write a claim function and a finish function, and they wire a two-line model adapter. The rule for this stretch is that no one is allowed to call the model until the ledger round-trips on fake data.

Minutes forty through sixty-five are the drill. The instructor calls out kills. Half the room loses its worker at item nine, the other half at item thirty, and everyone restarts with the identical command. Success is boring: the count of finished rows equals the count of inputs, and no output appears twice.

The last twenty-five minutes are the post-mortem and one extension. Students add a deadline, then argue about which failures should be retried and which should be parked for a human. The argument is the lesson.

The worked example

The ledger is three functions and one table. Nothing here is clever, which is the point — students should be able to explain every line out loud.

# ledger.py
import sqlite3, json, time, pathlib

DB = pathlib.Path("shift.db")

def connect():
    con = sqlite3.connect(DB, isolation_level=None)
    con.execute("PRAGMA journal_mode=WAL")
    con.execute("""CREATE TABLE IF NOT EXISTS jobs (
        key        TEXT PRIMARY KEY,
        input_sha  TEXT NOT NULL,
        state      TEXT NOT NULL,   -- running | done | failed
        attempt    INTEGER NOT NULL DEFAULT 0,
        output     TEXT,
        updated_at REAL NOT NULL
    )""")
    return con

The job key is derived from the input, not from the loop counter and not from a UUID. That single choice is what makes the restart safe: a resumed run regenerates the same keys and collides with the rows already on disk.

def claim(con, key, sha):
    con.execute("BEGIN IMMEDIATE")
    row = con.execute("SELECT state FROM jobs WHERE key=?", (key,)).fetchone()
    if row and row[0] == "done":
        con.execute("COMMIT")
        return False
    con.execute(
        """INSERT INTO jobs (key, input_sha, state, attempt, updated_at)
           VALUES (?, ?, 'running', 1, ?)
           ON CONFLICT(key) DO UPDATE SET
             state='running', attempt=attempt+1, updated_at=excluded.updated_at""",
        (key, sha, time.time()),
    )
    con.execute("COMMIT")
    return True

def finish(con, key, output):
    con.execute(
        "UPDATE jobs SET state='done', output=?, updated_at=? WHERE key=?",
        (json.dumps(output), time.time(), key),
    )

The worker itself is a loop with a clock attached. The deadline is not decoration; without it a slow endpoint turns a batch into an all-night process that the free tier may reclaim halfway through the last item.

# worker.py — pseudocode shape; the model call is your adapter
DEADLINE_S = 45 * 60

def run(items, budget_s=DEADLINE_S):
    con, start = connect(), time.monotonic()
    for key, sha, prompt in items:
        if time.monotonic() - start > budget_s:
            print("deadline reached, stopping cleanly")
            break
        if not claim(con, key, sha):
            continue                        # already done, skip
        try:
            finish(con, key, call_model(prompt, timeout=60))
        except Exception as exc:
            con.execute("UPDATE jobs SET state='failed', updated_at=? WHERE key=?",
                        (time.time(), key))
            print("failed", key, type(exc).__name__)

call_model is an adapter students fill in against whatever endpoint and model id their own account page shows. The workshop deliberately never hardcodes a model name, because half the confusion in these sessions comes from copying a string out of a blog post that was accurate a month ago.

The run command and the resume command are the same command. That is the punchline:

python worker.py --input tickets.jsonl
# instructor kills it here, or you just close the laptop
python worker.py --input tickets.jsonl

How students verify the drill honestly

A batch job that "looks finished" is worthless. Two queries close the loop, and both belong in the post-mortem rather than the happy path.

sqlite3 shift.db "select state, count(*) from jobs group by state;"
sqlite3 shift.db "select count(*) from jobs where state='done' and output is null;"

The first shows running, done, and failed side by side, so an abandoned running row cannot hide behind a cheerful log line. The second catches the quiet failure mode where a model returns an empty string, the worker marks the row done, and nobody notices until a week of labels turns out to be blank. Adding a schema check on the parsed output — one small function, ten lines — is the natural extension exercise for faster students.

Limitations, and who should skip this

The pattern costs a write before every model call and a read after every restart, which is fine for overnight batches and wrong for anything a user is waiting on. It also assumes the model call is deterministic enough that retrying a failed row does not silently change its meaning; when outputs are consequential, the failed row should park for a human instead of retrying.

Skip this workshop if you already run Temporal, Celery with a proper result backend, or any queue with at-least-once delivery and idempotency keys built in — you would be hand-rolling a worse version of a tool you already have. Skip it too if the workload touches regulated data, since a free-tier server is the wrong place to discover your retention policy. And skip it if the batch is under a hundred items run once; a shell script and a forgiving attitude is honestly enough.

For everyone else, the value is not the code. It is the twenty minutes where the room watches a process die mid-sentence, restarts it with the same command, and sees the totals line up. If you want to run the drill with a group, the operator of this account uses MonkeyCode's free model access and free server option for it; read the current program terms first, then go kill your own worker on purpose and count the rows.

Tags: ai, python, tutorial, opensource