Treat the Free Lane as an Evictable Cache

A free lane is a cache, not a contract. Cache it well and an eviction costs you a delay. Wire it as a dependency and an eviction costs you an outage.

I run a small batch summarizer over an internal corpus. It calls MonkeyCode's free model access and runs its worker on the free server option. Both are useful, and both are availability I did not buy. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Check the plan page yourself before you size anything, because free tiers move.

The constraint comes first

You cannot promise what you do not control. Free capacity can be rate limited, rotated, or retired. Assume revocation without notice, at the worst hour.

So the first design question is not which model. It is what your job does when the lane closes. If the answer is "it raises", you built a dependency. Fix that before you write a single prompt.

The second question is what your job does when the answer arrives twice. Retries are cheap on your side and expensive on theirs. Duplicate work is how a free lane becomes a slow lane.

Data flow, one direction

The path is short on purpose. A producer writes jobs to a local queue file. A worker reserves budget, calls the lane, writes the result to disk, then commits.

The worker runs on the free server, but it holds no state you cannot rebuild. Its ledger is a SQLite file it owns. Its dead-letter file is append-only. Kill the box and you lose a lease, not a decision.

That split matters. Anything that can be reclaimed must be disposable. Anything that must survive lives in a file you can copy off the box in one command.

tar czf rescue.tgz ledger.db dead_letter.jsonl out/

Failure domains, ranked

The interesting failures are not crashes. They are slow, partial, and silent. Here is the table I actually review before each change.

Failure	How you see it	Blast radius	First move
Rate limit	HTTP 429 on a burst	One job, then all jobs	Open the breaker for a cool-off window
Upstream error	5xx or reset mid-body	Retry storm	Cap retries, jitter the backoff
Allowance spent	Ledger refuses the reserve	Whole lane	Stop the queue, keep the exit
Server reclaimed	Health probe goes quiet	Worker only	Rerun from ledger, do not re-charge
Partial write	Result file is truncated	One job	Write to .tmp, rename after fsync
Clock skew	at values go backwards	Audit only	Store monotonic deltas, not wall time

The third row is the one people miss. Quota exhaustion is not an error path you sprinkle on top. It is a state the worker must enter cleanly and leave cleanly.

The lane as code

Two classes are enough. One holds the budget. One holds the exit. Everything else is a caller.

"""lane.py - one free lane, one ledger, one exit."""
import json, sqlite3, time, urllib.error, urllib.request


class LaneClosed(Exception):
    """The lane is unavailable right now. That is a state, not a bug."""


class Lane:
    def __init__(self, url, cap_tokens=200_000, db_path="ledger.db",
                 fails_to_open=3, cooloff=60.0, timeout=20):
        self.url, self.cap, self.db_path = url, cap_tokens, db_path
        self.fails_to_open, self.cooloff, self.timeout = fails_to_open, cooloff, timeout
        self.fails, self.open_until = 0, 0.0

    def _db(self):
        db = sqlite3.connect(self.db_path)
        db.execute("CREATE TABLE IF NOT EXISTS spend ("
                   "job_id TEXT PRIMARY KEY, tokens INTEGER NOT NULL, at REAL NOT NULL)")
        return db

    def spent(self):
        db = self._db()
        try:
            return db.execute("SELECT COALESCE(SUM(tokens),0) FROM spend").fetchone()[0]
        finally:
            db.close()

    def reserve(self, job_id, tokens):
        """Meter first. Idempotent per job_id. Returns False at the ceiling."""
        db = self._db()
        try:
            total = db.execute("SELECT COALESCE(SUM(tokens),0) FROM spend").fetchone()[0]
            done = db.execute("SELECT 1 FROM spend WHERE job_id=?", (job_id,)).fetchone()
            if done:
                return True
            if total + tokens > self.cap:
                return False
            db.execute("INSERT INTO spend VALUES (?,?,?)", (job_id, tokens, time.time()))
            db.commit()
            return True
        finally:
            db.close()

    def seen(self, job_id):
        db = self._db()
        try:
            return db.execute("SELECT 1 FROM spend WHERE job_id=?", (job_id,)).fetchone() is not None
        finally:
            db.close()

    def call(self, payload):
        if time.time() < self.open_until:
            raise LaneClosed(f"breaker open for {self.open_until - time.time():.0f}s")
        req = urllib.request.Request(
            self.url, data=json.dumps(payload).encode(),
            headers={"Content-Type": "application/json",
                     "Idempotency-Key": payload["job_id"]})
        try:
            with urllib.request.urlopen(req, timeout=self.timeout) as resp:
                self.fails = 0
                return json.loads(resp.read())
        except urllib.error.HTTPError as err:
            if err.code == 429 or err.code >= 500:
                self._trip(f"http {err.code}")
            raise
        except (urllib.error.URLError, TimeoutError) as err:
            self._trip(repr(err))
            raise

    def _trip(self, why):
        self.fails += 1
        if self.fails >= self.fails_to_open:
            self.open_until = time.time() + self.cooloff
        raise LaneClosed(why)

The caller stays boring, which is the point. The worker checks, reserves, calls, and defers.

"""run.py - spend the lane, keep the exit."""
import json, os
from lane import Lane, LaneClosed

lane = Lane(os.environ["LANE_URL"], cap_tokens=int(os.environ.get("LANE_CAP", "200000")))


def run(job):
    if lane.seen(job["job_id"]):
        return {"job_id": job["job_id"], "state": "already_done"}
    estimate = len(json.dumps(job).split()) * 2          # rough on purpose
    if not lane.reserve(job["job_id"], estimate):
        return {"job_id": job["job_id"], "state": "budget_stop"}
    try:
        out = lane.call(job)
        with open(f"out/{job['job_id']}.tmp", "w") as fh:
            fh.write(json.dumps(out))
        os.replace(f"out/{job['job_id']}.tmp", f"out/{job['job_id']}.json")
        return {"job_id": job["job_id"], "state": "done"}
    except LaneClosed as why:
        with open("dead_letter.jsonl", "a") as fh:
            fh.write(json.dumps({**job, "why": str(why)}) + "\n")
        return {"job_id": job["job_id"], "state": "deferred", "why": str(why)}

Prove the exit with a fake lane

You do not test eviction against a live endpoint. You test it against a server that misbehaves on demand. Three tests cover most of the risk.

"""test_lane.py - no network, no vendor, just the failure modes."""
import http.server, json, threading
import pytest
from lane import Lane, LaneClosed


class Fake(http.server.BaseHTTPRequestHandler):
    statuses = [200]

    def do_POST(self):
        code = Fake.statuses.pop(0) if Fake.statuses else 200
        self.rfile.read(int(self.headers.get("Content-Length", 0)))
        self.send_response(code)
        self.end_headers()
        self.wfile.write(json.dumps({"ok": code == 200}).encode())

    def log_message(self, *args):
        pass


@pytest.fixture
def lane(tmp_path):
    srv = http.server.HTTPServer(("127.0.0.1", 0), Fake)
    threading.Thread(target=srv.serve_forever, daemon=True).start()
    yield Lane(f"http://127.0.0.1:{srv.server_port}/v1/chat", cap_tokens=100,
               db_path=str(tmp_path / "l.db"), fails_to_open=2, cooloff=30)
    srv.shutdown()


def test_two_429s_open_the_exit(lane):
    Fake.statuses = [429, 429]
    for _ in range(2):
        with pytest.raises(LaneClosed):
            lane.call({"job_id": "a"})
    with pytest.raises(LaneClosed, match="breaker open"):
        lane.call({"job_id": "b"})


def test_budget_is_a_hard_stop(lane):
    assert lane.reserve("j1", 60) is True
    assert lane.reserve("j2", 60) is False


def test_reserve_is_idempotent(lane):
    assert lane.reserve("j1", 10) is True
    assert lane.reserve("j1", 10) is True
    assert lane.spent() == 10

Run it with two commands. There is nothing else to install beyond pytest.

python -m pytest -q test_lane.py
>>> 3 passed

Now point LANE_URL at the free endpoint and replay the same drill. If the breaker holds, you own the exit. If the worker crashes on a 429, you do not.

What I would change next

The single queue is my weakest decision. Bulk backlog and fresh jobs compete for the same allowance. Split them, and let the fresh lane borrow only when the bulk lane is idle.

I would also cache by content hash before spending anything. Repeats are common in a corpus, and a hit costs zero. Right now the ledger sees them as fresh work.

A warm standby key is the third change. One paid fallback behind the same interface turns a hard stop into a slower day. It is not free, and that is exactly why it works.

Finally, I would sample egress logs weekly. Prompts leave your box and land somewhere else. Count what left, and diff it against what you meant to send.

Who should skip this

Skip this design if you have a hard latency SLO. A breaker is honest, but it is not fast. Skip it for regulated data that cannot leave your network boundary at all.

Skip it if you need per-token billing you can reconcile to the cent. A free allowance is fine for experiments and poor for accounting. Skip it if your job needs hardware you cannot reclaim in one command.

For everything else, the pattern holds. Assume eviction, keep the ledger, and make the exit boring.

If you want to watch a lane close safely before you trust it, start on MonkeyCode's free model access and free server, then run the three tests above against it. A lane you can lose is a lane you can actually use.