48-Hour Field Notes: My Cron Job Was Idempotent Until the Container Got Replaced

I spent two days building a small daily reporting job, and the code was correct the whole time; the deployment was not. The job ran fine in my terminal, ran fine in a loop, and then ran twice in one day on a host I had rebuilt between runs. Why? Because I had put the thing that made it idempotent on the same disk as the thing that was disposable.

This is not a post about a platform deleting my data, and I want to be precise about that up front. I never verified any persistence guarantee for the free server option I was using, so I had no right to assume one. What I actually had was a mental model where "the file is still there" and "the process restarted" were independent facts, and in a throwaway host they are not.

What the job was supposed to do

The task itself is boring on purpose: once per UTC day, aggregate yesterday's rows, post one summary, write one receipt. The only interesting requirement is that a retry, a redeploy, or a double-scheduled tick must not produce a second summary. That requirement lives or dies on one question: where does the "I already did this" bit live?

My first answer was embarrassingly casual. I used a ./state/seen.json file next to the script, loaded it at start, appended the day key, and wrote it back. It passed every test I ran, because every test I ran happened to reuse the same working directory.

The assumption hiding in that file

Look at what that design quietly requires: the filesystem must outlive the process, the working directory must be the same on every start, and nothing else may write that path. In a long-lived VM all three are usually true, which is exactly why the bug never shows up in staging. In a host that gets replaced, the first one is false and the other two become irrelevant.

The reproduction: wipe the state, watch the duplicate

I replaced the JSON file with a SQLite store behind a tiny interface, so I could swap the "durable" location without touching job logic. SQLite is not magic either, but it gives me an atomic INSERT ... PRIMARY KEY claim, which is the primitive I actually wanted all along.

# job/store.py
from __future__ import annotations

import sqlite3
from contextlib import contextmanager
from pathlib import Path


class Store:
    """Dedupe store. The path must point somewhere the process does not own."""

    def __init__(self, path: str | Path) -> None:
        self.path = Path(path)
        self.path.parent.mkdir(parents=True, exist_ok=True)
        with self._conn() as conn:
            conn.execute(
                "CREATE TABLE IF NOT EXISTS receipts ("
                " key TEXT PRIMARY KEY,"
                " claimed_at TEXT NOT NULL)"
            )

    @contextmanager
    def _conn(self):
        conn = sqlite3.connect(self.path, timeout=5, isolation_level=None)
        try:
            yield conn
        finally:
            conn.close()

    def claim(self, key: str, now: str) -> bool:
        """True exactly once per key, even across process restarts."""
        with self._conn() as conn:
            try:
                conn.execute(
                    "INSERT INTO receipts (key, claimed_at) VALUES (?, ?)",
                    (key, now),
                )
                return True
            except sqlite3.IntegrityError:
                return False

The runner stays deliberately small, and the clock is an argument rather than a global. That single choice turned a vague "sometimes it double-posts" complaint into a test I can rerun in two seconds.

# job/run_once.py
import argparse
import datetime as dt
import json
import os
import sys

from job.store import Store


def window_key(now: dt.datetime, window: str = "day") -> str:
    if now.tzinfo is None:
        raise SystemExit("refusing to schedule on a naive timestamp")
    if window != "day":
        raise SystemExit(f"unknown window: {window}")
    utc = now.astimezone(dt.timezone.utc)
    return "daily-report:" + utc.strftime("%Y-%m-%d")


def main(argv=None) -> int:
    p = argparse.ArgumentParser()
    p.add_argument("--now", default=None, help="ISO-8601 with offset; tests inject this")
    p.add_argument("--store", default=os.environ.get("JOB_STORE", "./state/receipts.db"))
    args = p.parse_args(argv)

    now = dt.datetime.fromisoformat(args.now) if args.now else dt.datetime.now(dt.timezone.utc)
    key = window_key(now)

    if not Store(args.store).claim(key, now.isoformat()):
        print(json.dumps({"event": "skipped", "key": key, "reason": "already claimed"}))
        return 0

    print(json.dumps({"event": "ran", "key": key}))
    return 0


if __name__ == "__main__":
    sys.exit(main())

Now the tests. The third one is the important one, because it asserts the failure instead of the fix, and a test that documents the bug is worth more than a comment that apologizes for it.

# tests/test_idempotency.py
import shutil

import pytest

from job.run_once import main


def run(store, now):
    return main(["--store", str(store), "--now", now])


def test_second_tick_same_utc_day_is_a_no_op(tmp_path, capsys):
    store = tmp_path / "state" / "receipts.db"
    run(store, "2026-11-09T00:05:00+00:00")
    run(store, "2026-11-09T23:55:00+00:00")
    lines = capsys.readouterr().out.strip().splitlines()
    assert '"skipped"' in lines[1]


def test_tokyo_midnight_is_still_the_same_utc_day(tmp_path, capsys):
    store = tmp_path / "state" / "receipts.db"
    run(store, "2026-11-09T00:30:00+09:00")  # 2026-11-08T15:30Z
    run(store, "2026-11-09T10:00:00+00:00")
    lines = capsys.readouterr().out.strip().splitlines()
    assert '"ran"' in lines[1]


def test_replaced_host_replays_the_job(tmp_path, capsys):
    store = tmp_path / "state" / "receipts.db"
    run(store, "2026-11-09T00:05:00+00:00")
    shutil.rmtree(store.parent)  # simulate a fresh container with the same key
    run(store, "2026-11-09T00:05:00+00:00")
    lines = capsys.readouterr().out.strip().splitlines()
    assert '"ran"' in lines[1]  # the failure, made visible and repeatable


def test_naive_timestamp_is_rejected(tmp_path):
    with pytest.raises(SystemExit):
        run(tmp_path / "r.db", "2026-11-09T00:05:00")

python -m pytest -q tests/test_idempotency.py
JOB_STORE=/data/state/receipts.db python -m job.run_once --now "$(date -u +%FT%T+00:00)"

All three tests passed on the second run of the suite, which tells you nothing about the host and everything about the clock handling. That is the sobering part: my timezone bug and my durability bug looked identical from the outside, and only one of them was in the code.

Three rules I would keep from this exercise

1. Put the dedupe key somewhere the process cannot delete. A local file is fine for a scratch script, but treat any host-local path as cache, never as memory. If you cannot name the volume's lifetime, you cannot name the job's.
2. Inject the clock, and reject naive timestamps at the boundary. datetime.utcnow() in a scheduler is a bug with a delay fuse, and the fuse burns fastest for anyone running near midnight UTC.
3. Log a receipt per run, not a log line per step. I print one JSON object per invocation, ran or skipped, with the key included, so a duplicate is a one-line grep instead of an archaeology project.

A decision table for the state file

State	Keep on host disk?	Where it belongs
Dedupe key / run receipt	No	Durable store you can name
Last-seen cursor	No	Durable store, written after the side effect
Parsed config	Yes, rebuildable	Baked in or fetched at start
Model response cache	Yes, if loss is cheap	Local disk with a TTL
Credentials	Never	Environment or secret store

That table is the whole post in six rows. Everything I got wrong sat in the first two lines, and everything I got right sat in the middle two.

Where the free model actually fit

There is a second half to this job: after a successful run it writes a short human-readable digest of what changed overnight. I send the raw diff to a free model tier for that summary, then store the summary next to the receipt. Two things matter here, and both are boring on purpose.

• The model never decides whether the job runs; it only describes a run that already happened.
• The summary is regenerable, so it is cache, not state, and losing it costs one extra call.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used the free server option as the throwaway host for exactly this kind of experiment, and the free model access for the digest step, because both are things I am willing to lose mid-run while I watch what happens.

Limitations and who should skip this

This workflow will not save you if your side effect is not idempotent downstream, because a perfect dedupe key in front of a non-idempotent POST just moves the duplicate. It also assumes you have somewhere durable to put a row, and if you are running with zero external storage, your honest options are a single scheduled trigger or an at-least-once design with a visible duplicate. Finally, if your job must survive a hard kill mid-write, none of this replaces a real transaction boundary in the system that owns the data.

If you are running a one-shot script that nobody depends on, skip the receipt table and go outside instead. If anything downstream pages a human, the receipt is cheaper than the apology.

What I would repeat in the next 48 hours

The habit I am keeping is writing the failing test before the fix, and specifically a test that deletes the state directory to prove the duplicate exists. It takes eight lines, it fails for the right reason, and it converts a vague worry about hosts into a check I can run on every change. What would your version of that test delete, and would your job still be able to tell you it had already run?