Meter First, Call Second: A Ledger for Free Inference

Free inference does not fail on quality. It fails on accounting. You cannot subtract a number you never recorded.

Most teams wire a free lane in an afternoon. They add a key, a base URL, and a retry loop. Then a worker dies mid-call and the budget line goes quiet. Nobody knows what was spent, because nothing was written down.

This is an architecture review of the fix. Put a ledger in front of the lane. Record before you call, not after.

The constraints you actually have

You do not own the upstream. Quota can vanish between two requests, and the error text changes with it. You cannot plan capacity against a number you cannot read.

Workers are cheap and disposable. A process can die between the request and the response. Any budget held in memory dies with it, silently.

Retries are the default, not the exception. A timeout retry can double your spend on a call that already succeeded upstream. Your ledger has to survive that without human cleanup.

The free server option makes this sharper. Cold starts stretch tail latency, so timeouts fire more often. More timeouts mean more retries. That is the loop that eats a free allowance.

Data flow: reserve, call, settle

Keep one rule. No call without an open lease. The lease is a row, not a variable.

job ──▶ reserve(job_id, budget) ──▶ lease: open, ledger -budget
             │
             ├──▶ upstream lane (free model access)
             │
        commit(spent)  ──▶ ledger +(budget - spent)
        release()      ──▶ ledger +budget
        sweeper()      ──▶ release expired leases

The encumbrance is the sum of open budgets and committed spend. The ledger sum must always equal minus that number. If the two drift, you have a bug, not a rounding error.

The artifact: a ledger you can run today

Standard library only. One SQLite file. This runs on a laptop or a small always-on box.

# ledger.py
import sqlite3, time, uuid

DDL = """
CREATE TABLE IF NOT EXISTS lease (
  lease_id TEXT PRIMARY KEY,
  job_id   TEXT NOT NULL UNIQUE,
  budget   INTEGER NOT NULL CHECK (budget >= 0),
  spent    INTEGER NOT NULL DEFAULT 0,
  state    TEXT NOT NULL CHECK (state IN ('open','committed','released')),
  expires  REAL NOT NULL
);
CREATE TABLE IF NOT EXISTS ledger (
  entry_id INTEGER PRIMARY KEY AUTOINCREMENT,
  lease_id TEXT NOT NULL,
  delta    INTEGER NOT NULL,
  reason   TEXT NOT NULL,
  at       REAL NOT NULL
);
"""


class Broker:
    def __init__(self, path="budget.db", ttl=120.0, now=time.time):
        self.db = sqlite3.connect(path, isolation_level=None)
        self.db.execute("PRAGMA journal_mode=WAL")
        self.db.executescript(DDL)
        self.ttl, self.now = ttl, now

    def _tx(self, fn):
        self.db.execute("BEGIN IMMEDIATE")
        try:
            out = fn()
        except Exception:
            self.db.execute("ROLLBACK")
            raise
        self.db.execute("COMMIT")
        return out

    def _entry(self, lease_id, delta, reason):
        self.db.execute(
            "INSERT INTO ledger(lease_id,delta,reason,at) VALUES(?,?,?,?)",
            (lease_id, delta, reason, self.now()),
        )

    def reserve(self, job_id, budget):
        def run():
            row = self.db.execute(
                "SELECT lease_id, state FROM lease WHERE job_id=?", (job_id,)
            ).fetchone()
            if row:
                if row[1] != "open":
                    raise ValueError(f"job {job_id} is {row[1]}")
                return row[0]  # retry reuses the open lease
            lease_id = uuid.uuid4().hex
            self.db.execute(
                "INSERT INTO lease(lease_id,job_id,budget,expires) VALUES(?,?,?,?)",
                (lease_id, job_id, budget, self.now() + self.ttl),
            )
            self._entry(lease_id, -budget, "reserve")
            return lease_id
        return self._tx(run)

    def commit(self, lease_id, spent):
        def run():
            budget, state = self.db.execute(
                "SELECT budget, state FROM lease WHERE lease_id=?", (lease_id,)
            ).fetchone()
            if state != "open":
                return  # idempotent settle
            if spent > budget:
                self._entry(lease_id, -(spent - budget), "overrun")
            else:
                self._entry(lease_id, budget - spent, "refund")
            self.db.execute(
                "UPDATE lease SET spent=?, state='committed' WHERE lease_id=?",
                (spent, lease_id),
            )
        return self._tx(run)

    def release(self, lease_id):
        def run():
            budget, state = self.db.execute(
                "SELECT budget, state FROM lease WHERE lease_id=?", (lease_id,)
            ).fetchone()
            if state != "open":
                return
            self._entry(lease_id, budget, "release")
            self.db.execute(
                "UPDATE lease SET state='released' WHERE lease_id=?", (lease_id,)
            )
        return self._tx(run)

    def sweep(self):
        rows = self.db.execute(
            "SELECT lease_id FROM lease WHERE state='open' AND expires < ?",
            (self.now(),),
        ).fetchall()
        for (lease_id,) in rows:
            self.release(lease_id)
        return len(rows)


def encumbrance(broker):
    return broker.db.execute(
        "SELECT COALESCE(SUM(CASE state WHEN 'open' THEN budget ELSE spent END),0) FROM lease"
    ).fetchone()[0]


def ledger_sum(broker):
    return broker.db.execute("SELECT COALESCE(SUM(delta),0) FROM ledger").fetchone()[0]

Call it with a settle path that branches, not a bare finally. A finally commit would bill a failed call.

lease = broker.reserve(job_id="job-8241", budget=4000)
try:
    text = call_upstream(prompt, max_tokens=4000)
except Exception:
    broker.release(lease)      # upstream gave you nothing
    raise
else:
    broker.commit(lease, spent=count_tokens(prompt) + count_tokens(text))

Run the sweeper on a short interval. A crashed worker then costs you one TTL, not a frozen budget.

$ python -c "from ledger import Broker; print(Broker().sweep())"
0

The test that matters

Do not test the happy path. Test the invariant under failure. Use a fake upstream that fails on a fixed fraction of jobs.

import random
from ledger import Broker, encumbrance, ledger_sum

def fake_call(rand):
    if rand.random() < 0.3:
        raise TimeoutError("lane timeout")
    return rand.randint(1, 5000)   # simulated spent

def test_invariant_holds_under_failure():
    b, rand = Broker(":memory:", ttl=0.0), random.Random(7)
    for i in range(200):
        lease = b.reserve(f"job-{i}", 4000)
        try:
            spent = fake_call(rand)
        except TimeoutError:
            b.release(lease)
        else:
            b.commit(lease, spent)
    b.sweep()
    assert ledger_sum(b) == -encumbrance(b)

ttl=0.0 forces every open lease to expire immediately. The sweep then proves the release path works. A single drift means a code path skipped the ledger.

Failure domains, one by one

Failure	What you see	Containment
Worker dies after reserve	Budget stuck, no spend	TTL sweeper
Retry of a live job	Two leases for one job	job_id UNIQUE, reuse open lease
Response exceeds budget	spent > budget	Separate overrun entry, alert on it
Ledger write fails	Call runs unledgered	Reserve first; no row, no call
Upstream quota gone	429 storm, retry pileup	Per-lane breaker, queue instead of retry
Cold start on free server	Timeouts, then retries	Warm ping, cap queue depth

Read that table as a budget. You get to fix three of these today. Pick the two that touch money and the one that touches latency.

Where MonkeyCode fits

One honest lane for this. MonkeyCode offers free model access and a free server option, and the operator states a 10M-token free allowance as of publication. Free-tier numbers move, so check the current terms page before you size anything.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Use it for the parts the ledger protects. Run the broker on the free server and point one lane at free model access. Keep your paid lane behind the same broker. Then a free-lane outage is a routing event, not an incident.

What I would not do is put the broker in the same process as the worker. A crash takes both down. Separate the ledger, even onto the same box.

Who should not use this

If you have one user and one key, this is overhead. Your bill is small enough to read by hand.

If you cannot count tokens before the call, budget in requests instead. The schema works unchanged; the numbers get coarser.

If you need multi-region writes, SQLite is the wrong store. Keep the reserve/commit/release contract and move it to a transactional service.

This ledger records spend. It does not prevent an upstream from billing you anyway. Treat it as your source of truth, not as a firewall.

What I would change next

Move the reserve to the edge closest to the caller. A lease taken one network hop earlier survives more failures.

Add per-tenant budgets as a second column. One noisy job should not spend a shared pool.

Export the encumbrance as a metric. Alert when it sits flat while traffic is up. That is a leak, not a quiet period.

If you want to try the free model access and the free server, start with the ledger above. Point real jobs at the lane only after the invariant test passes twice.