The short version. A weekend project that calls a model endpoint from an ephemeral free server loses work every time the instance restarts mid-run. The fix below is an append-only JSONL journal keyed by a hash of the job spec, so a restart replays the file and skips finished work by key instead of by position. It is about sixty lines of Python, needs no database, and survives kill -9.
What actually broke
My runner processed a list of items in order and wrote results to a list in memory. Item 400 of 900 was in flight when the box recycled. Everything before item 400 was already done, but the process had no way to prove it, so the next run started at item 1.
Position-based resume fails for three reasons. The input order is not stable across restarts once you filter or re-sort. A retry after a partial write can duplicate a result. And a run that dies between "response received" and "result stored" is indistinguishable from a run that never reached that item.
A key-based journal answers a narrower and more honest question: has this exact job spec already produced a recorded outcome? If yes, skip. If no, redo it. That question survives crashes, reordering, and human edits to the job list.
Weekend scope
Kept (three things): an append-only journal on local disk, a spec hash as the idempotency key, and a replay path that is the same code as the first run.
Cut (five things): a Redis/Celery queue, a SQLite or Postgres result store, retry-with-backoff, a web UI, and any auth layer. Each cut is reversible later; none of them block a working demo.
I ran this on the free server option that MonkeyCode offers, against its free model access — the project lists a 10-million-token allowance at the time of writing, and you should re-read the current terms before sizing a build around that number. I did not benchmark throughput, latency, or uptime, and nothing below depends on a specific model name or hardware spec.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
The artifact: an append-only journal
One file, one record type, one rule — never rewrite an existing line. Torn trailing lines are tolerated on read because a crash can leave a half-written record.
# journal.py
import hashlib, json, os, time
from pathlib import Path
class Journal:
def __init__(self, path):
self.path = Path(path)
self.path.parent.mkdir(parents=True, exist_ok=True)
self.path.touch(exist_ok=True)
@staticmethod
def key(job):
blob = json.dumps(job, sort_keys=True, separators=(",", ":")).encode()
return hashlib.sha256(blob).hexdigest()[:16]
def _append(self, record):
line = json.dumps(record, sort_keys=True) + "\n"
with open(self.path, "a", encoding="utf-8") as f:
f.write(line)
f.flush()
os.fsync(f.fileno())
def state(self):
done, failed = {}, {}
for raw in self.path.read_text(encoding="utf-8").splitlines():
if not raw.strip():
continue
try:
rec = json.loads(raw) # a torn final line is skipped, not fatal
except json.JSONDecodeError:
continue
if rec.get("status") == "done":
done[rec["key"]] = rec
elif rec.get("status") == "failed":
failed[rec["key"]] = rec
return done, failed
def mark(self, key, status, **extra):
self._append({"key": key, "status": status, "ts": int(time.time()), **extra})
The fsync call is the part people drop. Without it the line lives in the OS page cache, and a hard restart can take the record with it. The cost is one syscall per job, which is irrelevant next to a network round trip.
The runner
Resume is not a separate mode. The first run and the thousandth run execute the same function; only the journal contents differ.
# run.py
from journal import Journal
from jobs import load_jobs
def call_model(job):
... # returns {"text": str, "usage": int} or None on empty response
def run(jobs, journal):
done, _ = journal.state()
for job in jobs:
k = Journal.key(job)
if k in done:
print(f"skip {k} (already recorded)")
continue
out = call_model(job)
if out is None:
journal.mark(k, "failed", reason="empty-response")
continue
journal.mark(k, "done", usage=out["usage"], preview=out["text"][:120])
if __name__ == "__main__":
run(load_jobs("jobs.json"), Journal("state/journal.jsonl"))
Two deliberate choices. Failed jobs are recorded, not silently retried, so a rerun does not loop forever on a permanently broken spec. And the stored preview is truncated, keeping the journal small enough to grep and cat during debugging.
If a failure was transient, delete the failed line and rerun. That is the entire recovery runbook, and it is why the format is plain text.
Testing crash-safety in ten minutes
The test plan is three commands, not a framework. Run all three before you trust the demo.
-
Partial run, hard kill. Start the runner over 50 jobs,
pkill -9 -f run.pyafter a few seconds, then rerun the same command. Expect the second pass to printskip ...lines and to touch only the missing keys. - Duplicate invocation. Launch the runner twice against the same journal while the first is still alive. Expect duplicated network calls, not corrupted state — the journal is append-only, so the later record wins on read. This is the known weakness of the design, covered below.
-
Torn line. Append
{"key": "deadbeto the file and rerun. Expect a clean start, because unparsable lines are skipped.
Count the done lines after each step with wc -l state/journal.jsonl. The number should match the number of unique spec hashes, not the number of attempts.
Journal, queue, or database
The journal is a resumability primitive, not a scheduler. Pick the smallest row that covers your actual failure mode.
| Failure mode | Journal enough? | Reach for |
|---|---|---|
| Instance restarts mid-run, single worker | Yes | JSONL journal |
| Two workers on the same job list | No | Lease/lock table with expiry |
| Need queryable history and totals | Partly | SQLite or Postgres |
| Downstream side effect must fire once | No | Idempotency key at the sink |
| Retries with backoff and dead-letter | No | Real queue |
The honest read of this table: the journal buys you exactly one property, and it buys it cheaply. Everything else is a different problem wearing the same clothes.
Where this design falls over
- Concurrent writers. Append-only protects against torn records, not against two processes doing the same work. On one free instance that is academic; on two, it is double billing.
- Journal growth. Reading the whole file on every start is fine at ten thousand lines and wasteful at a million. Rotate or move to SQLite past that point.
- Large payloads. Storing full output text turns the journal into a slow, unindexed document store. Store a pointer, not the body.
-
Free-tier churn. A recycled instance can take local disk with it. Copy
state/journal.jsonloff the box after every run, or the whole scheme is decoration.
Who should skip this
Skip it if you need exactly-once side effects in another system, run more than one worker, or already have a job runner with retries. Skip it if the workload is short enough to rerun from scratch in under a minute — the journal is pure overhead there. And skip it if you cannot persist the file somewhere durable, because a crash-safe log on a disk that vanishes is just a longer way to lose data.
Close-out
The weekend deliverable was a demo that resumes after a hard kill, with a scope cut list I could defend. The journal is the only piece I would keep in a real system; the rest was scaffolding to get something running on free infrastructure before Monday.
If you want to reproduce it, the artifact is one file and three shell commands. MonkeyCode's free access and free server option are what let me run the loop without a billing conversation — worth a try if your weekend budget is zero, and worth reading the current terms before you depend on them.
Top comments (0)