Your job wrote the file. You redeployed. The file was gone.
So which part failed: the write, or your assumption about the disk?
The bug shape you will hit
- A scheduled job appends rows to
state.jsonl. - The container restarts for an unrelated reason.
- The file comes back empty, missing, or half-written.
- The job re-runs and double-sends, double-charges, or double-builds.
Nothing in that list is exotic. Free compute just makes it routine.
Myth 1: "A restart is a clean reset"
Restarting a process clears memory. It does not clear disk.
That cuts both ways. Your cache and in-memory locks disappear, which is fine. Your stale worker.lock, your half-finished .part file, and yesterday's build artifact may not.
A restart is a partial reset you never specified. Good to know before you debug it at 2am.
Evidence: write a marker file, restart, read it back. That is the whole test. We automate it below.
Myth 2: "/tmp is temporary, the app dir is permanent"
Folder names are not durability policies.
On plenty of container hosts, the app directory is an image layer. A redeploy rebuilds it. Your "permanent" file was never on a volume; it was on a layer that gets replaced.
/tmp may be a tmpfs. It may also be the same overlay as everything else. The name tells you nothing.
Corrected model: ask where the bytes physically land, not what the folder is called.
Myth 3: "If the write returned, the data is safe"
Three boundaries exist here, and they fail separately.
-
write()plusflush()moves bytes from your process to the OS page cache. -
os.fsync()pushes them toward the storage device. - The volume itself can be discarded on the next deploy or scale event.
Steps 1 and 2 protect you from a crash. Nothing in your code protects you from step 3.
Myth 4: "Free compute is only for demos"
Wrong in one direction, right in another.
Free compute can run real jobs, as long as state lives outside it and every job is idempotent. What it cannot be is your database, your queue, or your source of truth.
Treat it as disposable compute and it behaves predictably. Treat it as a server and it will surprise you.
The probe: a survival check
Two modes. write appends a checksummed record. verify re-reads and re-hashes every line.
#!/usr/bin/env python3
"""survival_check.py -- does this path survive a restart?"""
import argparse, hashlib, json, os, time, uuid
from pathlib import Path
def target(state_dir: Path) -> Path:
return state_dir / "survival.jsonl"
def digest(rec: dict) -> str:
body = json.dumps(rec, sort_keys=True)
return hashlib.sha256(body.encode("utf-8")).hexdigest()
def cmd_write(state_dir: Path, note: str) -> None:
state_dir.mkdir(parents=True, exist_ok=True)
rec = {"id": uuid.uuid4().hex[:12], "note": note, "ts": round(time.time(), 3)}
rec["sha256"] = digest(rec)
with open(target(state_dir), "a", encoding="utf-8") as f:
f.write(json.dumps(rec, sort_keys=True) + "\n")
f.flush()
os.fsync(f.fileno()) # survives a crash, not a redeploy
print(f"wrote id={rec['id']} to {target(state_dir)}")
def cmd_verify(state_dir: Path) -> int:
path = target(state_dir)
if not path.exists():
print(f"MISSING {path}") # the path did not survive
return 2
bad, n = 0, 0
for n, line in enumerate(path.read_text(encoding="utf-8").splitlines(), 1):
rec = json.loads(line)
claimed = rec.pop("sha256", None)
ok = claimed == digest(rec)
bad += 0 if ok else 1
print(f"line={n} id={rec['id']} checksum={'ok' if ok else 'MISMATCH'}")
print(f"lines_checked={n} mismatches={bad}")
return 0 if bad == 0 else 1
if __name__ == "__main__":
ap = argparse.ArgumentParser()
ap.add_argument("mode", choices=["write", "verify"])
ap.add_argument("--dir", required=True, type=Path)
ap.add_argument("--note", default="probe")
a = ap.parse_args()
raise SystemExit(cmd_verify(a.dir) if a.mode == "verify" else cmd_write(a.dir, a.note))
Write to three candidate paths, then restart the process or container:
for d in /tmp/survival ./state /data/survival; do
python3 survival_check.py write --dir "$d" --note before-restart \
|| echo "WRITE FAILED $d"
done
# restart the process, container, or instance here
for d in /tmp/survival ./state /data/survival; do
echo "== $d"
python3 survival_check.py verify --dir "$d"
done
Reading the results
| verify output | what it means | what to do |
|---|---|---|
MISSING |
the path vanished | move state to a real volume or external store |
MISSING but write succeeded |
write went to a layer that was discarded | stop trusting that path |
mismatches > 0 |
bytes changed or truncated | suspect the storage layer, not your hash |
all ok
|
survived this restart | repeat after a redeploy before believing it |
One clean pass proves very little. A restart is not a redeploy. A redeploy is not a scale event. Run the probe after each one you can trigger on purpose.
Then make the job idempotent
Persistence is half the fix. The other half is a key.
def run_once(job_key: str, store) -> bool:
if store.seen(job_key): # store must live outside the free box
return False
store.mark(job_key)
do_the_work(job_key)
return True
If store is a file on the ephemeral disk, you renamed the bug. You did not fix it.
Where free model access fits
I use a model for two narrow jobs here: drafting the checksum logic, and reviewing idempotency keys for collisions. Both are text problems with testable answers, which is where a model earns its keep.
MonkeyCode is an open-source project offering free model access and a free server option. That is enough to run the write → restart → verify loop above without paid infrastructure. The exact free allowance changes over time, so read the current terms on the project page instead of trusting a number quoted in any article, including this one.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
Do not read "it ran on free compute" as a durability claim. Read it as evidence about one boundary, on one day.
Who should skip this
- You need a written durability or uptime guarantee. Use a managed database and a paid runtime.
- Your data carries residency or audit requirements a shared free host cannot meet.
- Your job moves money or sends messages. Get a real queue first.
- You cannot redeploy on demand. Then you cannot test the boundary that matters.
The one-line version
A restart is a memory reset, not a disk reset. Probe the disk, externalize the state, key the job. Then trust it, after you have re-run the probe once more.
Want to try the loop yourself? The free server option is a reasonable place to start.
Top comments (0)