48-Hour Field Notes: I Built a Boot Beacon Because My Free Server Never Said Goodbye

My batch job kept its progress in a Python dict, and I genuinely believed that was fine for a two-hundred-item run that should take eleven minutes. Then the output file came back with more lines than the input file, and three of them were near-identical summaries of the same ticket. Nothing in the log said "restart" — no traceback, no exit code, no farewell note from the process. So the real question was not "why did the model return the same thing twice?" but "how many times did my process actually start?"

The next 48 hours were mostly me learning to answer that second question before touching anything else. This is what I tried, what broke, what I would repeat, and the small amount of code that survived the weekend.

The bug looked like a duplicate-row bug, and it wasn't

My first instinct was the usual one: assume the endpoint answered twice and the writer appended twice. I added a seen set, deduplicated at the end, and declared victory around hour two. The victory lasted exactly one more restart, because a set in memory has the same lifetime problem as a dict in memory — it forgets everything the moment the interpreter goes away.

Here is the uncomfortable part. My worker had no durable record of starting, so I could not distinguish three very different events:

1. The host rebooted and took the filesystem with it.
2. The container restarted, keeping the same kernel but a new process tree.
3. My own code caught an exception, re-entered the loop, and re-processed the tail of the list.

Each of those needs a different fix, and I was debugging all three at once with no evidence. Does that sound familiar?

Step 1: Make the process announce every start

The cheapest instrument I have ever added is a boot beacon: one append-only JSONL file, written once at startup and once at shutdown. It records the kernel boot id, the process start time in clock ticks, the pid, and a random instance label.

# boot_beacon.py
import json, os, time, uuid
from pathlib import Path

BEACON = Path("run/boot.jsonl")

def boot_id() -> str:
    try:
        return Path("/proc/sys/kernel/random/boot_id").read_text().strip()
    except FileNotFoundError:
        return "unknown"

def proc_start_ticks() -> int:
    # Field 22 of /proc/self/stat: starttime in clock ticks since boot
    stat = Path("/proc/self/stat").read_text()
    return int(stat.rsplit(") ", 1)[1].split()[19])

def record(event: str, **extra) -> None:
    BEACON.parent.mkdir(parents=True, exist_ok=True)
    row = {
        "ts": round(time.time(), 3),
        "event": event,
        "run_id": os.environ.get("RUN_ID", "local"),
        "pid": os.getpid(),
        "boot_id": boot_id(),
        "proc_start": proc_start_ticks(),
        "instance": str(uuid.uuid4())[:8],
        **extra,
    }
    with BEACON.open("a") as fh:
        fh.write(json.dumps(row) + "\n")

Before record("start") at the top of the worker, I assumed restarts were rare and correlated with deploys. After it, I could count them in one jq call: jq -r '.event + " " + .instance' run/boot.jsonl | sort | uniq -c.

Why the boot id alone is not enough

Experienced operators will spot the flaw immediately: inside a shared kernel, boot_id is identical across container restarts. That is exactly why I log proc_start too, because it changes on every new process even when the kernel never blinked. The pair turns a vague "something restarted" into a taxonomy you can act on.

boot_id	proc_start	Meaning
changed	changed	Host reboot; assume the local disk may be gone
same	changed	Process or container restart; files usually survive
same	same	Same process — this is a logic bug, not a restart

The third row is the one that saved me from rewriting a worker that was never being restarted at all.

Step 2: Move the state into a three-state ledger

A dict cannot survive a restart, but a SQLite file usually can, and this is where I stopped improvising. Every item gets exactly one row and moves through three states: claimed, called, committed. The rule I care about most is that the result row and the state flip happen inside one transaction, so a crash can never leave a written result without a matching commit.

import sqlite3, time
from pathlib import Path

DB = Path("run/ledger.db")

SCHEMA = """
CREATE TABLE IF NOT EXISTS jobs (
  item_id  TEXT PRIMARY KEY,
  state    TEXT NOT NULL,
  attempts INTEGER NOT NULL DEFAULT 0,
  receipt  TEXT,
  updated  REAL NOT NULL
);
CREATE TABLE IF NOT EXISTS results (
  item_id TEXT PRIMARY KEY,
  label   TEXT NOT NULL
);
"""

def connect():
    DB.parent.mkdir(parents=True, exist_ok=True)
    con = sqlite3.connect(DB)
    con.execute("PRAGMA journal_mode=WAL")
    con.executescript(SCHEMA)
    return con

def claim(con, item_id: str) -> bool:
    cur = con.execute(
        "INSERT INTO jobs(item_id, state, updated) VALUES(?, 'claimed', ?) "
        "ON CONFLICT(item_id) DO NOTHING", (item_id, time.time()))
    con.commit()
    return cur.rowcount == 1

def mark_called(con, item_id: str, receipt: str) -> None:
    con.execute(
        "UPDATE jobs SET state='called', receipt=?, attempts=attempts+1, updated=? "
        "WHERE item_id=? AND state='claimed'", (receipt, time.time(), item_id))
    con.commit()

def commit(con, item_id: str, label: str) -> None:
    with con:  # result row and state flip land together
        con.execute(
            "INSERT INTO results(item_id, label) VALUES(?, ?) "
            "ON CONFLICT(item_id) DO UPDATE SET label=excluded.label", (item_id, label))
        con.execute(
            "UPDATE jobs SET state='committed', updated=? "
            "WHERE item_id=? AND state='called'", (time.time(), item_id))

Notice what the receipt buys me. If the process dies between mark_called and commit, the next start finds rows in called and can finish them from the stored response instead of paying for a second call. That is the whole trick: the expensive step is bracketed by two cheap durable writes.

Step 3: Make the side effect idempotent before blaming the model

I wasted a few hours assuming the model was the unreliable component, and it was not. The unreliable component was my assumption that an append to a file is a safe place to record "done". With the results table above, re-processing is harmless because the primary key absorbs the repeat, and the export step becomes a plain query you can run any time: SELECT item_id, label FROM results ORDER BY item_id.

This is also where MonkeyCode enters the picture, and I want to be precise about what I mean. The preset for this run was MonkeyCode's free model access for the classification call and its free server option for hosting the worker, both of which are operator-supplied availability claims rather than things I benchmarked here.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What I will not tell you is how long that server keeps your process alive, because I do not know and I refuse to design around a guess. The beacon exists precisely so you can measure your own host's behavior in a day instead of arguing about it in a thread.

Step 4: Break it on purpose with a chaos harness

The most useful thirty lines I wrote all weekend kill my own worker in the middle of a batch. Run this twice — once against the naive dict version to watch it misbehave, once against the ledger version — and the difference is obvious in under a minute.

set -euo pipefail
rm -rf run out
python -c "from worker import connect; connect()"

RUN_ID=chaos-1 python worker.py --items 200 &
WPID=$!
sleep 4
kill -9 "$WPID" || true
wait "$WPID" 2>/dev/null || true

RUN_ID=chaos-1 python worker.py --items 200
echo '--- ledger states ---'
sqlite3 run/ledger.db "SELECT state, COUNT(*) FROM jobs GROUP BY state;"
echo '--- duplicates ---'
python - <<'PY'
import sqlite3
rows = sqlite3.connect("run/ledger.db").execute(
    "SELECT item_id, COUNT(*) c FROM results GROUP BY item_id HAVING c > 1").fetchall()
print("duplicate result keys:", len(rows))
PY
jq -r 'select(.event == "start") | .instance' run/boot.jsonl | wc -l

My acceptance criteria for this pattern are boring and strict, which is the point of writing them down before the run:

1. duplicate result keys is 0.
2. Every row in jobs ends in committed; none stay in claimed or called.
3. The beacon shows at least two distinct instance values for one run_id.
4. Re-running the same command a third time makes no new model calls at all.

If criterion 4 fails, you have an in-memory check somewhere that the ledger does not know about, and that is worth finding now rather than at 2 a.m.

Who should not use this

This pattern costs maybe forty lines and one extra table, but it is not free of trade-offs and it is not for everyone. Skip it when the script is genuinely one-shot and you can re-run it without anyone noticing. Skip it when your side effect cannot be made idempotent and you have no reconciliation query, because a ledger will not save you from a double email; you need a reconciliation job first. Skip it if you plan to run many workers in parallel against the same SQLite file, since a single writer is a feature here, not a bug.

And do not put the ledger on a network filesystem and expect WAL mode to behave, because it will not.

What I would repeat, and what I would change

I would repeat the beacon on day one, before the first long run, since it converted an argument into a jq query. I would repeat the three-state ledger, but I would write the called recovery path at the same time as the happy path instead of after the first confusing crash. What I would change is the order of investigation: instrument first, deduplicate second, and only then start suspecting the endpoint.

If you want to try this against a free model endpoint without renting anything, the free server option is a reasonable place to put a worker for an afternoon — just bring the beacon, and let the log tell you what your host really does with your process.