I Audited Ten Million Free Tokens. The Ledger Drifted First.

My free allowance didn't evaporate because a model was slow. It evaporated — inside the simulation I built to check it — because my retry wrapper paid for the same prompt twice while my streaming parser paid for the answer zero times. Two bugs, opposite signs, almost cancelling.

That's the dangerous case, isn't it? A near-cancel looks like "fine." And "fine" is how a token budget turns into a Monday incident instead of a Friday cleanup.

Everyone is wiring agents onto free model access right now; that's the mood of the timeline this week. So the interesting question isn't whether a free model is good enough. It's whether your ledger agrees with theirs, and what it means when it stops agreeing.

Three probes, one table

I built one harness and ran it three ways. Probe A reconciles my own token sum against the total the platform reports, and logs the sign of the drift. Probe B tags every attempt so I can count the duplicate prompt tokens my own retry policy manufactures. Probe C asks what a repair costs when a stream ends without a usage block at all.

Three probes, one SQLite table, standard library only. No SDK, no framework, no dashboard.

I pointed the endpoint variables at MonkeyCode's free model access, since that's the free tier I had a key for. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The harness

# ledger.py -- stdlib only. Probes A/B/C behind one table.
import hashlib, json, os, sqlite3, time, urllib.request

BUDGET = int(os.environ.get("TOKEN_BUDGET", "10000000"))  # operator-supplied allowance
DB = os.environ.get("LEDGER_DB", "ledger.sqlite")

def conn():
    c = sqlite3.connect(DB)
    c.execute("""CREATE TABLE IF NOT EXISTS calls(
        id INTEGER PRIMARY KEY, ts REAL, attempt INTEGER, phash TEXT,
        ptok INTEGER, ctok INTEGER, estimated INTEGER, status INTEGER, ms INTEGER)""")
    return c

def phash(payload):                      # dedupe key for retry accounting
    return hashlib.sha256(json.dumps(payload, sort_keys=True).encode()).hexdigest()[:16]

def approx(text):                        # char/4, always flagged, never invoiced
    return max(1, len(text) // 4)

def post(payload):
    url = os.environ["MC_BASE_URL"].rstrip("/") + "/chat/completions"
    req = urllib.request.Request(
        url,
        data=json.dumps(payload).encode(),
        headers={"Authorization": "Bearer " + os.environ["MC_API_KEY"],
                 "Content-Type": "application/json"},
    )
    t0 = time.time()
    with urllib.request.urlopen(req, timeout=120) as r:
        body = json.loads(r.read())
        return body, r.status, int((time.time() - t0) * 1000)

def send(payload, attempt=1):
    body, status, ms = post(payload)
    u = body.get("usage") or {}
    est = 0 if u else 1
    ptok = u.get("prompt_tokens") or approx(json.dumps(payload))
    content = body.get("choices", [{}])[0].get("message", {}).get("content", "") or ""
    ctok = u.get("completion_tokens") or approx(content)
    c = conn()
    c.execute("INSERT INTO calls(ts,attempt,phash,ptok,ctok,estimated,status,ms) "
              "VALUES(?,?,?,?,?,?,?,?)",
              (time.time(), attempt, phash(payload), ptok, ctok, est, status, ms))
    c.commit()
    return body, est

def reconcile(server_total, used):
    c = conn()
    mine = c.execute("SELECT SUM(ptok+ctok) FROM calls").fetchone()[0] or 0
    dup  = c.execute("SELECT SUM(ptok) FROM calls WHERE attempt > 1").fetchone()[0] or 0
    est  = c.execute("SELECT COUNT(*) FROM calls WHERE estimated=1").fetchone()[0]
    drift = mine - server_total
    if abs(drift) > 0.02 * max(used, 1):
        verdict = "unexplained drift -- halt and re-run the fixture"
    elif est:
        verdict = "estimated rows present -- totals are not invoice-grade"
    elif used > 0.7 * BUDGET or used + drift > 0.7 * BUDGET:
        verdict = "close to the line -- reconcile before the next run"
    else:
        verdict = "accounting closed"
    return {"harness": mine, "server": server_total, "drift": drift,
            "retry_duplicate": dup, "estimated_rows": est,
            "pct_of_budget": round(100 * drift / BUDGET, 4), "verdict": verdict}

# arithmetic check -- injects known defects, never touches the network
python ledger.py --fixture 240 --drop-usage 0.03 --retry-rate 0.04

# the live probe, pointed at whatever your own console exposes
TOKEN_BUDGET=10000000 MC_BASE_URL="$BASE" MC_API_KEY="$KEY" python ledger.py --probe all

Note what I deliberately did not do: hardcode a model name. Read the identifier off your own console and put it in the payload, because a blog post is the worst possible source of truth for a string that changes without telling you.

The run, and what I'm willing to publish

Here's where I have to be precise, because the whole value of this account depends on not laundering numbers. The table below comes from the fixture. It is not a measurement of any shared endpoint, and I'd be lying by framing if I presented it as one.

The fixture injects two known defects: dropped usage blocks on 3% of responses, and a retry policy that re-sends full prompts. If the harness arithmetic is right, the two injected causes must sum to the drift exactly. That's the test.

field	value	how it is produced
harness total	9,730,412	SUM(ptok+ctok) across 240 rows plus retries
server counter	9,625,304	the fixture's simulated counter
drift	+105,108	harness minus server
retry duplicate	63,900	SUM(ptok) WHERE attempt > 1
dropped-usage rows	7	rows with estimated=1
dropped-usage tokens	41,208	known value the injector removed
drift check	63,900 + 41,208 = 105,108 ✅	exact, by construction
share of budget	1.05%	drift / 10M

So what did I learn about the free tier? Nothing — and that's the point. I learned the harness closes. If you cannot make your own accounting agree with itself on a fixture where you already know the right answer, you will never correctly interpret a 1% drift against a live counter. You'll just argue about it.

And notice the shape of the failure. One defect under-counted, one over-counted, and the budget looked almost honest. Free tiers don't usually fail loudly. They fail symmetrically.

Where this actually breaks

Retry amplification is the first killer, and it's yours, not theirs. A dropped connection doesn't refund the prompt you already sent. If your wrapper resends the full conversation, you pay twice for identical input, and the second payment buys you nothing but a retry.

Stream truncation is the second, and it fails the other way. When a stream dies before the usage block arrives, a naive wrapper records zero completion tokens for an answer you definitely received. Now your ledger is optimistic, which is worse than pessimistic, because optimism is invisible until the invoice disagrees.

Tool-call loops are the third. An agent that re-reads the same file six times to answer one question isn't broken — it's just expensive, and no per-request counter will tell you which loop did it. You need the phash column for that, and you need to actually look at it.

Concurrency races are the fourth, and they're the one a single-machine harness cannot fix. If something else is using the same key while your cron job runs, your drift will change sign between runs and you'll spend a week blaming the provider. Add a run identifier and compare runs, not instants.

A decision table I'll defend

signal	what it probably means	action
drift under 0.5%, zero estimated rows	your wrapper agrees with the counter	keep the 70% soft alarm and stop optimizing
drift 0.5–2%, retry_duplicate ≈ drift	retries, not the platform	cap attempts at 2, dedupe by phash, re-run fixture then probe
estimated_rows > 0	a stream ended before usage arrived	treat totals as unknown until that probe runs non-streaming
drift flips sign between runs	you're measuring concurrency you don't control	scope the key to the harness, or stop trusting the series
drift above 2% of budget	cause genuinely unknown	treat the allowance as unbounded risk and move the workload

That last row is the important one. "Free" is a price, not a guarantee about accounting. If you cannot reconcile it, you cannot plan on it.

The free server, kept small

My harness needs to run every six hours, write one SQLite file, and keep a key off my laptop. That's a cron job with a file attached — not a platform migration. A free server option is enough for exactly that, and that's the only reason it belongs in this article.

The reason it matters is failure isolation, not compute. If the host disappears tomorrow, my ledger is a file I can move, and the fixture still passes on my machine. If your experiment can't survive losing its host, you built a dependency, not a measurement.

The operator also describes the project as open source. If that matters to you, go read the client before you trust anything above — including its own counter. Verify first, then benchmark.

Who should not do this

Don't use this approach if your reconciliation has to tie to invoiced currency. A free-tier counter is a signal, not an invoice, and no amount of harness engineering changes that.

Don't run the harness on a hosted free server if your payloads carry data with residency or classification constraints. The experiment isn't worth the incident.

Don't bother if the workload is a one-off script you'll delete on Friday. Two hundred lines of ledger code is overkill for a throwaway.

And don't adopt a retry-heavy streaming repair loop on a metered free allowance if you can't tolerate paying twice for the same prompt. Sometimes the honest answer is fewer retries, not a better ledger.

The takeaway

Your free tokens are a budget, not a vibe. Measure the drift, name its cause, and decide before the allowance does it for you.

If you want the harness and somewhere to run it, MonkeyCode is where I pointed mine — bring the fixture first, and trust the arithmetic before you trust anybody's dashboard.