Tokens Outlive the Free Server

The crash is not the interesting part. The second after it is.

Your free server can vanish mid-request. The model call it started does not vanish with it. That asymmetry is the whole architecture problem, and most small deployments ignore it.

You run two lanes. One sits on a server you control loosely. The other sits on a provider you control not at all. They fail in different ways, at different times, with different costs.

Two failure domains, one request

A free server is evictable. A model call is consumed. When the server dies first, the work still finishes on the far side. You just never see the result.

Flip the order and it is harmless. If the model call fails, your server survives and retries. That direction is easy to reason about.

So design for the hard direction. Persist the intent before the call. Persist the outcome after it. Treat the gap between them as a known unknown, not as a bug.

Here is the flow you want.

client
  └─> free server ──(1) write intent ──> ledger (off-box)
        └─(2) call model ──> provider
              └─(3) write outcome ──> ledger

Step 2 can outlive the process that started step 1. That is the entire point of the diagram.

Order beats cleverness

Two wrong implementations show up again and again. The first writes the ledger after the call returns. It loses everything at exactly the moment you need the record.

The second writes only on error. It handles timeouts and misses crashes, which is the case you actually care about.

Do neither. Write first, then call, then write again. The first record is cheap and the second is boring.

One more rule. Persist the outcome even if the client has already disconnected. The tokens are spent. The answer is still worth keeping.

The ledger that survives eviction

Local disk on a free server is a cache. Never a system of record. Write the ledger somewhere that outlives the box, or accept that you are guessing.

Here is a minimal recorder. Standard library only, append-only, fsync on every write.

# lane_ledger.py
import hashlib, json, os, time, uuid

LEDGER = os.environ.get("LEDGER_PATH", "ledger.jsonl")

def _append(record: dict) -> None:
    line = json.dumps(record, separators=(",", ":")) + "\n"
    fd = os.open(LEDGER, os.O_WRONLY | os.O_CREAT | os.O_APPEND, 0o644)
    try:
        os.write(fd, line.encode("utf-8"))
        os.fsync(fd)  # an eviction right after this write must not lose it
    finally:
        os.close(fd)

def begin(prompt: str) -> str:
    rid = uuid.uuid4().hex
    _append({"k": "intent", "id": rid, "ts": time.time(),
             "prompt_sha": hashlib.sha256(prompt.encode()).hexdigest()})
    return rid

def finish(rid: str, status: str) -> None:
    _append({"k": "outcome", "id": rid, "ts": time.time(), "status": status})

Call begin, make the request, call finish. If the process dies in between, the intent stays orphaned. That orphan is information, not noise.

Note what is not in the record. The prompt body is hashed, not stored. A ledger is a second egress path and deserves the same suspicion as the first.

Reconcile at boot

On every start, scan for intents without outcomes. Do not retry them silently. You cannot tell whether the provider already consumed the allowance.

def reconcile(path: str = LEDGER) -> list[str]:
    seen, done = {}, set()
    with open(path, encoding="utf-8") as f:
        for line in f:
            rec = json.loads(line)
            if rec["k"] == "intent":
                seen[rec["id"]] = rec
            elif rec["k"] == "outcome":
                done.add(rec["id"])
    return sorted(set(seen) - done)

Log the count as a metric at startup. A rising orphan count means your timeout is shorter than your tail latency. That is a tuning signal you get for free.

Reproduce it without spending anything

Do not test this against a real endpoint first. Point the worker at a local stub that sleeps longer than your client timeout.

# stub_model.py — sleeps, then answers
import json, time
from http.server import BaseHTTPRequestHandler, HTTPServer

class H(BaseHTTPRequestHandler):
    def do_POST(self):
        self.rfile.read(int(self.headers["Content-Length"]))
        time.sleep(5)
        body = json.dumps({"output": "ok"}).encode()
        self.send_response(200)
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

HTTPServer(("127.0.0.1", 8787), H).serve_forever()

Now run the sequence. Kill the worker while the stub is still asleep.

python stub_model.py &
python lane_worker.py --endpoint http://127.0.0.1:8787 &
sleep 1 && pkill -9 -f lane_worker.py
python -c "import lane_ledger; print(lane_ledger.reconcile())"

You should see exactly one id. That id is your orphan. Run it five times and the count should be five.

Read the failures carefully. Zero orphans means your fsync is missing. Two orphans per run means you are writing the intent twice. A clean number is the only proof you have.

The whole test costs nothing and takes about ten seconds.

What belongs where

State placement decides whether a restart is boring or expensive.

State	Home	Reason
Intent record	off-box, append-only	local disk dies with the box
Prompt text	hash on the ledger, body elsewhere	the ledger is a second egress path
Idempotency key	sent to the provider	a retry must not double-consume
Rendered answer	short-TTL cache	the far side may answer after the client left
Counters	derived from the ledger	in-memory counters reset on eviction

Two rules cover most of it. Anything you would miss after a restart goes off-box. Anything you would not miss is a cache.

Limitations, honestly

This pattern costs writes. Every request becomes two durable appends. On a free server with slow storage, that is real latency you pay on every single call.

It also does not solve delivery. Reconciliation tells you an orphan exists. It cannot recover the response. If the answer matters, the client has to poll a job record instead of holding a connection open.

Skip this if your workload is fire-and-forget. Skip it too if you cannot place the ledger off-box. A local ledger on an evictable disk is theater.

Try it on two lanes

You can rehearse this cheaply. MonkeyCode exposes free model access and a free server option, so both lanes are available to break on purpose. Check the current limits on the site before you size anything around them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Build the stub first. Kill the worker on purpose. Count the orphans. Then deploy.