What does "done" mean when the process that writes your results can disappear at any second? I spent two days on a batch extraction job that printed a clean zero in the pending column while the results table underneath it was quietly short a few rows. The job called a model I reached through MonkeyCode's free model access, and the worker itself ran on the free server option that comes with the account. Disclosure: This article was prepared as part of MonkeyCode's product outreach.
The disclosure is the least interesting part of this post, so let me get to the bug. I had two separate writes per item, no transaction wrapping them, and a counter that read the wrong one of the two. If you have ever watched a queue hit zero and felt relief before checking the output, this story is for you.
Hour 0-6: the counter that lied
My worker was a plain loop: claim an item, call the model, write the payload, mark the item done. The pending count came from the item table, and the item table was always the last thing I touched. So when the box recycled underneath me, the counter was the most accurate-looking number in the system and the least trustworthy one.
Here is the shape of what I had, and yes, it is as thin as it looks:
item = db.claim_next() # status -> 'done' happens inside here
payload = call_model(item)
db.save_result(item.id, payload)
Can you see the hole? If the process dies between claim_next() and save_result(), the item is permanently marked done with nothing behind it. There is no retry path, no error, no alarm. The queue drains, the dashboard turns green, and the missing rows ship to whoever reads the export next.
Hour 6-18: two writes, no transaction
The fix is not clever. It is the boring one: make completion a single atomic act that includes the result. I moved everything into SQLite with WAL mode enabled and a busy timeout, because the free box had enough memory for that and I did not want to babysit a separate database.
CREATE TABLE items (
id TEXT PRIMARY KEY,
input_hash TEXT NOT NULL,
prompt_version TEXT NOT NULL,
status TEXT NOT NULL DEFAULT 'pending',
lease_owner TEXT,
lease_expires_at REAL
);
CREATE TABLE results (
idem_key TEXT PRIMARY KEY,
item_id TEXT NOT NULL,
payload TEXT NOT NULL,
created_at REAL NOT NULL
);
The write became one transaction: insert the result with INSERT OR IGNORE, then flip the item to done. If the insert conflicts, I already have the answer and the flip is a no-op. If the process dies mid-transaction, SQLite rolls back and the item stays claimable. That single change removed the silent-loss failure mode entirely.
The claim needs a lease, not a flag
A boolean in_progress flag is a promise you make to yourself and break on the next restart. I replaced it with a lease: a claim sets an owner and an expiry, and the claim query is happy to steal expired leases back.
CLAIM = """
UPDATE items
SET status='claimed', lease_owner=?, lease_expires_at=?
WHERE id IN (
SELECT id FROM items
WHERE status='pending'
OR (status='claimed' AND lease_expires_at < ?)
ORDER BY id LIMIT ?
)
RETURNING id, input_hash, prompt_version;
"""
The RETURNING clause needs SQLite 3.35 or newer, which is worth checking before you copy this into an old base image. A heartbeat thread extends the expiry every few seconds while the model call is in flight, so a long call does not get stolen out from under a healthy worker.
Hour 18-30: the restart duplicated calls
With leases in place, my restarts stopped losing work and started duplicating it. The same item came back, the model answered again, and because generation is not deterministic, I got two different payloads for one input. That was worse than the original bug in one specific way: my agreement check compared samples, so duplicated-but-divergent answers made the quality gate flap between pass and fail across runs.
This is the part people skip. Exactly-once delivery is not something you get from a queue; it is something you get from an idempotency key that the worker derives itself. Mine is a hash of the input plus the prompt version, not a random UUID handed out at claim time, because a random key from a retried worker gives you two keys and two rows.
idem_key = sha256(f"{input_hash}:{prompt_version}".encode()).hexdigest()
Hour 30-42: the key needed the prompt version
I originally hashed only the input, and that cost me an afternoon. I edited the extraction prompt, re-ran the batch, and every item hit the cache and returned the old payload without a single model call. The job looked fast because it was doing nothing.
Putting prompt_version in the key means a prompt edit invalidates old rows on purpose, while a plain retry after a crash still hits the same key and reuses the stored answer. If you want the opposite behavior, hash the input alone and version your results table instead — just decide deliberately rather than discovering it at 2 a.m.
Hour 42-48: the kill test I would repeat
The only test that mattered was the one where I murdered the worker mid-call. Not a graceful shutdown, not a timeout, a hard SIGKILL at a random moment, repeated until I stopped being surprised.
# 20 rounds: start, kill -9 after a random delay, then assert invariants
for i in $(seq 1 20); do
python worker.py & pid=$!
sleep $(awk -v s=$RANDOM 'BEGIN{printf "%.2f", s/32768}')
kill -9 $pid; wait $pid 2>/dev/null
python check_invariants.py --db batch.sqlite || { echo "round $i failed"; exit 1; }
done
echo "all rounds clean"
The invariant script checks three things and nothing else: no item is done without a matching row in results; no idem_key appears twice; every claimed item either has a live lease or has been returned to pending. Write the assertions in plain SQL and print the offending rows, because a bare assert False in a loop tells you nothing about which invariant you broke.
A decision table for the free path
Availability claims like "free models" and "free server" are not the same as a guarantee, and I had to be honest with myself about where each one fits. This is the table I now fill in before starting a batch job:
| Situation | Free server + free models? | Why |
|---|---|---|
| Overnight batch, results read tomorrow | Yes | Restarts only cost time, and the lease makes time recoverable |
| Anything a user is waiting on, sub-second | No | You cannot promise a latency budget you do not control |
| Data you are not allowed to send off-box | No | Check your obligations before the first request, not after |
| Experiments where the model choice may change | Yes | Prompt and model version belong in the idempotency key anyway |
| Production writes to a system of record | Not directly | Land results in your own store, then promote deliberately |
Notice that none of those rows are about price. The free tier was not the interesting variable; the interesting variable was whether my code could survive a restart without lying to me.
Who should not use this approach
If you need an uptime commitment, a fixed concurrency ceiling, or a compliance story about where inference happens, this workflow is the wrong tool and no amount of retry logic fixes that. If your team will not run the kill test in CI, do not adopt the lease pattern either — an untested lease is just a more complex flag. And if your batch is small enough to finish in one sitting with a human watching, the plain loop is fine; the overhead only pays for itself once restarts are routine.
What I would repeat on Monday
I would keep the leased claim, the single-transaction completion, the prompt version inside the idempotency key, and the 20-round kill loop in CI. I would drop the custom dashboard counter entirely and read completion from the results table, because the counter was never the source of truth and I kept treating it like one.
The uncomfortable lesson is that my first version looked correct for hours at a time, which is exactly long enough to trust it. If you want a place to rehearse that failure on purpose, the free model access and free server option on MonkeyCode are where I ran mine — bring the kill test with you, and tell me which invariant broke first.
Top comments (0)