48-Hour Field Notes: The Free Server Forgot Everything by Morning

At 03:12 the triage worker came back up and immediately re-answered six tickets it had already answered. Nobody had asked it to. The redeploy was routine. The memory of the previous one was not there.

That is the failure mode cheap hosting hands you. Not a crash — a quiet amnesia. So for this round of 48-hour field notes I gave myself one narrow question: what has to be true before I let a box I don't own hold state that matters?

The three checkpoints

I gave the experiment two days and three checkpoints. First, detect whether the filesystem survives a restart at all. Second, replay whatever survived without duplicating side effects. Third, keep the boot honest when the model that summarises the log is the slowest thing in the path.

That third one matters more than it sounds. A cold start plus a model call is two clocks stacked on top of each other, and the second clock is the one you cannot see.

Checkpoint one: write a canary, then go to sleep

The first thing I built was not a service. It was a witness. A twelve-line probe that writes a canary file at boot, fsyncs it, and reports on the next boot whether it is still there.

# probe_boot.py — run this as the very first thing in your start command
import json, os, pathlib, time, uuid

STATE_DIR = pathlib.Path(os.environ.get("STATE_DIR", "/tmp/appstate"))
CANARY = STATE_DIR / "canary.json"

def fsync_write(path: pathlib.Path, text: str) -> None:
    path.parent.mkdir(parents=True, exist_ok=True)
    fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o644)
    try:
        os.write(fd, text.encode())
        os.fsync(fd)
    finally:
        os.close(fd)

def check() -> dict:
    if CANARY.exists():
        prev = json.loads(CANARY.read_text())
        return {"durable": True, "previous_boot": prev["boot"], "age_s": round(time.time() - prev["at"], 1)}
    boot = uuid.uuid4().hex[:12]
    fsync_write(CANARY, json.dumps({"boot": boot, "at": time.time()}))
    return {"durable": False, "previous_boot": None, "boot": boot}

if __name__ == "__main__":
    print(json.dumps(check()))

Illustrative output on a box whose disk does not survive a restart — this is the shape to look for, not a measurement from your machine:

{"durable": false, "previous_boot": null, "boot": "9f3c1a77b204"}

If you run that every boot for a day and durable never flips to true, you are not looking at a filesystem. You are looking at a scratch pad that gets wiped. Everything downstream of that sentence changes.

Checkpoint two: the journal, and why it is append-only

The fix is not a database. It is a decision about who owns the truth. The box owns execution. Something durable owns the record. Between them sits an append-only journal with a unique key per side effect.

# journal.py
import json, sqlite3, time

SCHEMA = """
CREATE TABLE IF NOT EXISTS journal (
  seq        INTEGER PRIMARY KEY AUTOINCREMENT,
  key        TEXT UNIQUE NOT NULL,
  payload    TEXT NOT NULL,
  created_at REAL NOT NULL
);
"""

def connect(path: str) -> sqlite3.Connection:
    con = sqlite3.connect(path, isolation_level=None)
    con.execute("PRAGMA journal_mode=WAL")
    con.execute("PRAGMA synchronous=FULL")
    con.executescript(SCHEMA)
    return con

def record(con, key: str, payload: dict) -> bool:
    try:
        con.execute(
            "INSERT INTO journal (key, payload, created_at) VALUES (?,?,?)",
            (key, json.dumps(payload, sort_keys=True), time.time()),
        )
        return True
    except sqlite3.IntegrityError:
        return False  # already recorded; do not act twice

The caller asks one question before doing anything expensive: has this key already been recorded? If the answer is yes, skip. If the process dies halfway through, the journal is still the arbiter and the replay is boring. Boring is the goal.

Note the two pragmas. WAL plus synchronous=FULL is slower than the default and I would still take it here, because the whole point of this file is to survive the exact moment the machine stops existing.

Checkpoint three: two clocks, one budget

Cold starts are easy to eyeball and hard to argue about. Argue with a loop instead.

for i in $(seq 1 20); do
  curl -s -o /dev/null -w '%{time_starttransfer}\n' "https://$HOST/healthz"
  sleep 45
done | tee starttransfer.txt

sort -n starttransfer.txt | head -3   # the tail you actually care about

Run it long enough to include one idle window. The number you want is not the median, it is the first request after silence. That request is the one your users hit at 07:00.

Once you have that number, the model call goes behind it, not in front of it. On the free model access side of MonkeyCode I use a cheap extraction pass to pull the failing predicate out of a CI log, then a verifier to confirm the predicate actually appears in the raw text before the answer is allowed out. The extractor is allowed to be wrong. The verifier is not.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The free model access and the free server option are availability claims supplied by the project, not capacity guarantees — quotas and hardware change, so read the current project page rather than trusting any number quoted in a blog post, including this one.

What the 48 hours actually bought

A table, mostly. It is the only artifact I trust after two days.

Workload	Safe on a rotating disk?	What to add
Stateless request/response	Yes	Nothing
Anything with a queue	No	Durable journal + idempotency key
In-process cron	No	External trigger, or accept double-fires
Local cache of model output	No	Rebuild on boot, never trust it
Long-running stream	Measure first	Reconnect budget, not retry storms

Two rows did the damage in my run: the in-process scheduler and the output cache. Both felt like optimisations. Both were state wearing a disguise.

Here is what I would repeat. Write the canary before writing the service. Make the journal the first file you create and the last thing you delete. And treat "the free tier is fine" as a hypothesis with a measurement attached, not a mood.

Who should skip this

If your workload is a long-lived WebSocket fan-out, a build runner that needs sustained CPU for twenty minutes, or anything holding a multi-gigabyte index in memory, stop reading and rent a real machine. The harness will only tell you, politely and repeatedly, that you already knew.

Skip it too if you cannot own a durable store somewhere else — a managed Postgres, an object bucket, a laptop that stays awake. The journal pattern without a durable home is just a more elaborate way to lose data.

And if your team has never run a post-mortem on a duplicate side effect, fix that habit before you fix the infrastructure. The duplicate ticket at 03:12 was not a hosting problem. It was a design that assumed the machine would remember.

If you want to run the canary and the journal against a box you don't own yet, the free server option is a cheap way to get your own numbers in an afternoon. Bring the journal. The server will not bring it for you.