48-Hour Field Notes: Every Restart Restarted My Batch Job, So I Added Leases

Two days into a batch classification run, I opened the log and saw line one again. Not a continuation of line one — the same line one, with a new timestamp and a new process id. How long does your job actually run before something outside your code decides it should start over?

That question is the entire article. A batch job that finishes on your laptop is a very different animal from a batch job that runs on a host you do not control, and the difference is not throughput. It is durability: whether the work you already paid for in time and tokens survives a restart you never scheduled.

What the 48-hour run was supposed to prove

My plan was boring on purpose: classify a few thousand short text records, one request per record, write the label into a table, and let it run overnight. I used MonkeyCode's free model access as the classification backend and ran the worker on the free server option, both of which the operator of this account tells me are available. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Confirm the current limits and terms in your own account, because I am deliberately not quoting quotas, hardware, or durations I cannot verify for you.

The 48 hours were not about model quality. They were about answering three questions with evidence instead of vibes:

1. Does the worker process survive between my runs, or does something replace it?
2. If it is replaced, does the local filesystem survive with it?
3. After a restart, does the job resume, or does it redo work it already completed?

I want to be explicit about one thing before any numbers would normally appear: I am not going to quote timings or counts from my own run, because you cannot reproduce my account, my data, or my moment in time. The probe below prints your numbers, on your machine, and that is worth more than my anecdote.

Failure mode 1: the process is not the job

The first thing that breaks is a category error in my own head. I had been treating "the script is running" as a property of the job, when it is only a property of one process. If the platform recycles that process, the job did not pause — from the job's point of view, it never existed.

The boot probe that tells you the truth

Drop this file next to your worker and let it run at the top of main(). It writes a boot record and compares it to the previous one.

# probe.py — answers "did I get a fresh process or a fresh machine?"
import json, os, pathlib, socket, time, uuid

STATE = pathlib.Path(os.environ.get("PROBE_STATE", "/tmp/probe_state.json"))

boot = {
    "boot_id": uuid.uuid4().hex,
    "pid": os.getpid(),
    "hostname": socket.gethostname(),
    "started_at": time.time(),
    "cwd": os.getcwd(),
    "tmp_writable": os.access("/tmp", os.W_OK),
}

try:
    prev = json.loads(STATE.read_text())
except (FileNotFoundError, json.JSONDecodeError):
    prev = None

STATE.write_text(json.dumps(boot))

print(json.dumps({
    "boot": boot,
    "previous_record_found": prev is not None,
    "process_replaced": prev is not None and prev["boot_id"] != boot["boot_id"],
    "state_survived_restart": prev is not None,
}, indent=2))

Run your worker three times and read the output. If previous_record_found is false every single time, your /tmp is not surviving restarts, and neither is anything else you wrote there.

Here is the part that took me a while to accept: from inside the process, a wiped disk and a brand new machine look identical. You cannot distinguish them with local evidence, which is exactly why the probe has to write somewhere that outlives the box.

Failure mode 2: at-least-once quietly becomes twice

Once I stopped trusting the process, I over-corrected and just reran the whole batch. That is how I generated duplicate work: same input, same prompt, same output, twice in the table. Would you notice that in a dashboard? Probably not — the row count goes up, and row count is a terrible consistency check.

My earlier instinct was to hash the inputs and call it done. Hashing gives you a key, but a key alone cannot tell you whether the request was in flight when the process died. For that you need a state machine.

-- ledger.sql — the smallest state machine that survives a restart
CREATE TABLE IF NOT EXISTS jobs (
  idem_key    TEXT PRIMARY KEY,
  payload     TEXT NOT NULL,
  state       TEXT NOT NULL CHECK (state IN ('pending','inflight','done','failed')),
  attempts    INTEGER NOT NULL DEFAULT 0,
  lease_until REAL,
  result      TEXT,
  updated_at  REAL NOT NULL
);

The key is a hash over canonical JSON of the payload and the request parameters — the prompt text, temperature, and the endpoint alias you configured, not a model name you might silently change later. Store this table outside the worker host, because a table on an ephemeral disk is a comment, not a ledger.

Claim, work, commit — in that order

now = time.time()
lease = 120.0

row = db.execute("""
    UPDATE jobs
       SET state = 'inflight', attempts = attempts + 1,
           lease_until = :now + :lease, updated_at = :now
     WHERE idem_key = (
           SELECT idem_key FROM jobs
            WHERE state = 'pending' ORDER BY updated_at LIMIT 1)
       AND state = 'pending'
    RETURNING idem_key, payload, attempts
""", {"now": now, "lease": lease}).fetchone()

if row is None:
    pass  # nothing claimable; reconciliation or exit handles it
else:
    idem_key, payload, attempts = row
    try:
        label = classify(payload)          # the free model call
    except Exception as exc:
        db.execute("UPDATE jobs SET state='pending', lease_until=NULL, "
                   "updated_at=? WHERE idem_key=? AND attempts < 5",
                   (time.time(), idem_key))
    else:
        db.execute("UPDATE jobs SET state='done', result=?, lease_until=NULL, "
                   "updated_at=? WHERE idem_key=?",
                   (label, time.time(), idem_key))

Notice what is not happening: there is no retry storm, because a failed attempt goes back to pending with a cap, and there is no duplicate write, because idem_key is a primary key. Write the result first, then flip the state — if you flip first and crash, the label is lost forever and the row claims to be done.

Failure mode 3: crashed work stays "inflight" forever

The nastiest failure is the quiet one. A process claims a job, dies mid-request, and the row sits in inflight with a lease that expires into nothing. Your queue looks busy and makes zero progress. Isn't that the perfect bug — one that keeps your monitoring green?

The fix is a reconciliation pass at every boot, before the worker starts claiming:

UPDATE jobs
   SET state = 'pending', lease_until = NULL, updated_at = :now
 WHERE state = 'inflight' AND lease_until < :now;

Make it idempotent, run it twice in your tests, and confirm the second run changes zero rows. That single assertion caught a bug in my first version, where I was recomputing updated_at inside the WHERE clause and re-queueing rows I had just finished.

My 48-hour protocol, if you want to repeat it

1. Hour 0: seed the ledger with pending rows and record the total count somewhere external.
2. Hour 0: start the worker with the probe at the top of main(), logging boot_id to an external sink.
3. Hour 6: kill the process hard with kill -9 and confirm reconciliation recovers the inflight row.
4. Hour 12: restart the host and check the probe output for a new boot_id and a wiped /tmp.
5. Hour 24: run a duplicate-detection query — SELECT idem_key, COUNT(*) FROM results GROUP BY 1 HAVING COUNT(*) > 1 — and expect zero rows.
6. Hour 48: compare done + failed against your external seed count, not against a local counter.

# expected: one line per boot, with changing boot_id and a fresh lease reclaim
python worker.py 2>&1 | tee -a /dev/stderr | jq -c '{boot: .boot.boot_id, replaced: .process_replaced}'

Decision table: is this worth the complexity?

Your situation	Use leases + ledger?	Why
Long batch on a free or shared host	Yes	Restarts are expected, not exceptional
Short job that finishes in one process	No	The ledger costs more than the rerun
Billing or counting per item	Yes	Duplicates become money, not noise
Sub-second interactive path	No	Lease bookkeeping adds latency you cannot spend
Anything with a strict exactly-once contract	No, not this	This design is at-least-once with dedupe on write

Who should not use this approach

If your job reliably finishes inside one process on a host you control, stop reading and go ship it — leases are overhead. If you need true exactly-once semantics for billing, an idempotency key plus a primary key is necessary but not sufficient, and you should talk to someone who owns that guarantee. And if your data cannot legally leave your network, a hosted model endpoint is the wrong backend regardless of how tidy your ledger is.

What I would repeat: the probe first, the ledger second, and the reconciliation pass before anything else touches the queue. What I would skip: rerunning the batch "just to be safe", which is how I manufactured my own duplicates in the first place.

If you want a low-stakes place to run the probe and the ledger end to end, the free model access and free server option are a reasonable sandbox for exactly this kind of experiment — just keep your counts external and your expectations honest.