Free Tokens Are a Cost Center Until You Track the Failures

Every release week brings another model, another token giveaway, and another benchmark that quietly contradicts the one before it. The current debate about what AI badges and token counts actually measure is useful, but it stops one step short of the question that matters. The question is not whether a free tier is generous; it is whether you can detect the failures it produces before your users do.

You land a free token allowance and a free server, and for a moment the AI budget problem looks solved. Then the first silent failure ships: a summary that drops the main conclusion, an extraction that returns valid JSON with the wrong field, a classification that picks the second-best label. Nobody notices until a user files a ticket, and by then the free tier has cost you an afternoon of archaeology and a measurable slice of trust.

My position is simple: free model access does not remove cost, it relocates it. Token prices move from the billing dashboard to engineering time, retry loops, and support queues, and most teams never see that second bill. The unit you should optimize is not tokens per dollar but engineering minutes per successful task, and the only way to optimize it is to record it.

The ledger beats the dashboard

A billing dashboard shows you what the provider charged, which is increasingly close to nothing. It does not show you the three retries behind a successful call, the manual correction that followed a plausible-but-wrong answer, or the user who churned because a summary buried the call to action. A failure-cost ledger captures exactly those hidden line items, and once you see them, the free tier stops being a marketing number and becomes an engineering input.

The ledger is not a benchmark suite and it is not a drift detector. A benchmark tells you how a model performs on someone else's data, and a drift detector tells you when the output distribution changes. The ledger tells you what a model costs you per finished task on your own workload, which is the only number that survives contact with production.

Step 1: Define success before you measure it

You cannot price failures until you can name them, so start with a small decision table for each task type. Keep the criteria mechanical enough that a script can apply them. If a criterion needs a human judgment call, the ledger will inherit that ambiguity and your numbers will become arguments instead of measurements.

Task	A successful result is...	A failed result looks like...
Summarize	At least three sentences, no "N/A", key entity present	One-line summary, hallucinated entity, missing conclusion
Extract	Valid JSON with all required fields populated	Valid JSON with the wrong field, or invalid JSON after retries
Classify	Label inside the allowed set with confidence above threshold	Out-of-vocabulary label, or correct label with low confidence

Write these down before you run a single call. If you write them after seeing the outputs, you will design criteria that make the model look good, and the ledger will be worthless.

Step 2: Log every call, including the ugly ones

Append one JSON line per call, with the task type, the status, the retry count, and the manual minutes you spent fixing the output. The manual minutes field is the one people skip, and it is the one that turns a cost ledger into a decision tool. Without it, you are measuring model quality, not the cost of running the model.

{"ts": "2026-08-21T02:11:00Z", "task": "extract", "status": "ok", "tokens": 1240, "retries": 2, "manual_minutes": 0, "output": "{\"name\": \"Acme\", \"plan\": \"pro\"}", "required_fields": ["name", "plan"]}
{"ts": "2026-08-21T02:14:00Z", "task": "summarize", "status": "ok", "tokens": 980, "retries": 1, "manual_minutes": 6, "output": "The report covers Q2 revenue. N/A", "required_fields": []}

The second line is the important one: the call succeeded, the tokens were nearly free, and it still cost you six minutes of a human's attention. That is the cost your dashboard will never show you. Multiply that by a hundred calls a day and the free tier starts looking like a subscription you pay in focus.

Step 3: Compute cost per successful task

The script below reads the JSONL log, applies your success criteria, and prints the metrics that matter. It is deliberately small so you can read every line and adapt it to your own task definitions. It treats the token price as zero because that is what the free allowance charges, and it forces the labor cost to carry the argument.

import json
import sys

def is_success(entry):
    if entry.get("status") != "ok":
        return False
    task = entry.get("task")
    output = entry.get("output", "")
    if task == "summarize":
        sentences = [s for s in output.split(". ") if s.strip()]
        return len(sentences) >= 3 and "N/A" not in output
    if task == "extract":
        try:
            data = json.loads(output)
            return all(k in data for k in entry.get("required_fields", []))
        except json.JSONDecodeError:
            return False
    if task == "classify":
        return output in entry.get("allowed_labels", [])
    return False

def summarize(path, labor_rate_per_minute=1.0):
    entries = [json.loads(line) for line in open(path) if line.strip()]
    total = len(entries)
    succeeded = sum(1 for e in entries if is_success(e))
    failed = total - succeeded
    tokens = sum(e.get("tokens", 0) for e in entries)
    retries = sum(e.get("retries", 0) for e in entries)
    manual_minutes = sum(e.get("manual_minutes", 0) for e in entries)
    labor_cost = manual_minutes * labor_rate_per_minute
    token_cost = tokens / 1_000_000 * 0.0  # free tier price
    per_success = (labor_cost + token_cost) / max(succeeded, 1)
    print(f"tasks={total} success={succeeded} failed={failed}")
    print(f"retries={retries} manual_minutes={manual_minutes}")
    print(f"cost_per_successful_task=${per_success:.4f}")

if __name__ == "__main__":
    summarize(sys.argv[1])

Run it against a week of logs and you will see the real shape of the free tier. A model that fails on ten percent of calls but needs five minutes of cleanup each time is more expensive than a paid model that fails on two percent and needs none. That comparison is the whole point of the ledger.

Step 4: Run it nightly on a free server

A ledger only works if it runs without you, so schedule it and let it accumulate. The server does not need a GPU or a big instance; it needs a cron daemon and a filesystem. A cron job on a free server is enough for this workload, and the eval calls themselves can come from a free token allowance without touching your production budget.

0 2 * * * cd /opt/ledger && python failure_ledger.py logs/$(date -d yesterday +\%F).jsonl >> reports/daily.txt

If you want a place to run this without provisioning a paid box, MonkeyCode's open-source project offers a free server option and a 10-million-token free allowance that covers exactly this kind of nightly evaluation. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I am not claiming the free tier is production-grade for high-volume traffic; I am claiming it is sufficient for a scheduled job that writes a few dozen lines to a file.

Limitations: who should not use this

The ledger is a measurement tool, not a quality guarantee, and it has real limits. It only knows the failures you define in step 1, so subtle quality decay that your criteria cannot express will slip past it. It also assumes a human is available to record manual minutes; if your team skips that field, the output is theater.

Teams with very low volume will see noisy numbers, and teams with strict compliance requirements should not run customer data through a free server at all. If you cannot define success mechanically, fix that first, because no ledger can price a failure you cannot name.

The next time someone announces a free token allowance, do not ask how many tokens it includes. Ask what it costs you when the output is wrong, and then go measure that number. If you want to try the ledger against a real free tier, MonkeyCode's free server and token allowance are a reasonable place to start.