Benchmark Inside the Quota Envelope

A benchmark score is comparable only to another score produced under the same resource envelope, and a metered free tier is an envelope like any other. When an agent run exhausts a token allowance halfway through a task, most harnesses record that task as failed and average it with everything else. The resulting mean mixes capability with bookkeeping, so the headline number describes the budget as much as the model. This article sets out a small harness discipline that keeps those two things separable, and every artifact below is labelled illustrative because none of it has been executed against a live provider here.

The failure mode hides inside a try/except

The common shape of the bug is a broad exception handler that converts any provider-side refusal into a task verdict. A quota rejection and a genuinely wrong answer land in the same column, and the summary line reports one number for two different events.

# Illustrative only: this pattern destroys the distinction we need.
def run_task(task, client):
    try:
        return client.solve(task.prompt)
    except Exception:
        return {"passed": False}   # quota, timeout, and wrong answer all collapse here

Once that collapse happens, no downstream statistic can recover the lost signal, because the information was discarded at the boundary where it was still cheap to keep. The repair costs three fields and one enum, and it has to be applied before the first graded run rather than after the matrix is complete. Retrofitting a ledger onto finished runs means rerunning them, which is exactly the spend the ledger was supposed to protect.

Estimate spend before the first graded task

A pilot over a fixed sample of the task file yields a spend estimate precise enough to reject an infeasible plan early. The estimator below projects from either the pilot mean or the pilot maximum, applies a safety factor, and refuses to schedule a matrix that cannot finish inside the declared budget.

# Illustrative only — not executed; substitute your own token accounting.
def estimate(sample_spend, total_tasks, budget, safety=1.25, central="max"):
    spend = sorted(sample_spend)
    if not spend:
        raise ValueError("pilot produced no usable traces")
    unit = spend[-1] if central == "max" else sum(spend) / len(spend)
    projected = unit * total_tasks * safety
    return {
        "unit_tokens": round(unit, 1),
        "projected_tokens": round(projected, 1),
        "budget_tokens": budget,
        "fits": projected <= budget,
        "headroom": round(budget - projected, 1),
    }

The pilot must sample from the same task file the matrix will use, and the sample should be drawn with a recorded seed so the estimate can be reproduced later. Taking the maximum instead of the mean is the conservative choice, because agent tasks have a long right tail and the tail is what exhausts an allowance. A projection built on the mean looks comfortable right up until a handful of long tasks consume the remaining headroom.

# Illustrative commands for a repository laid out as harness/ plus tasks.jsonl.
python -m harness.pilot --tasks tasks.jsonl --sample 24 --seed 7 --out pilot.json
python -m harness.plan --pilot pilot.json --tasks tasks.jsonl --budget 10000000 --out plan.json

The --budget value is a placeholder for whatever allowance the operator actually holds, including a free-tier grant; the number matters only because it has to match the envelope the results will be reported under. If plan.json returns fits: false, the correct response is to shrink the task set or the per-task cap rather than to run and truncate quietly. Truncation produces rows that look like ordinary failures and cannot be distinguished after the fact.

Record the stop reason on every row

The ledger is the only durable artifact of a benchmark, and it should be complete enough that the score can be recomputed without rerunning anything. Five stop reasons cover the cases that matter: completed, step cap, token cap, rate limited, and error. A validation pass catches missing rows and over-cap spend before anyone looks at a mean.

# Illustrative only.
STOP_REASONS = {"completed", "step_cap", "token_cap", "rate_limited", "error"}

def validate(records, planned_tasks):
    assert len(records) == planned_tasks, "ledger is missing task rows"
    for r in records:
        assert r["stop_reason"] in STOP_REASONS, r["task_id"]
        assert r["tokens_in"] + r["tokens_out"] <= r["token_cap"], r["task_id"]
    return {reason: sum(r["stop_reason"] == reason for r in records)
            for reason in STOP_REASONS}

A ledger that passes validation still says nothing about quality on its own, but it makes the next section possible and lets a reviewer replay the arithmetic without trusting the author. That replayability is the property separating a benchmark from a screenshot, and it is cheap to add while the harness is still small. The schema also makes an honest postmortem possible when a run finishes far under the projected spend.

Publish two numbers and one rate

The honest summary of a capped run reports the score across all planned tasks, the score across completed tasks only, and the censoring rate that connects them. The first behaves as a lower bound on what the configuration would do with unlimited budget; the second shows behaviour on the tasks that reached their own conclusion.

# Illustrative only.
def summarize(records):
    graded = [r for r in records if r["stop_reason"] == "completed"]
    passed = sum(bool(r["passed"]) for r in records)
    spend = sum(r["tokens_in"] + r["tokens_out"] for r in records)
    return {
        "score_all_tasks": passed / len(records),
        "score_completed_only": (sum(bool(r["passed"]) for r in graded) / len(graded)) if graded else None,
        "censoring_rate": 1 - len(graded) / len(records),
        "tokens_per_success": spend / passed if passed else None,
    }

Reporting both numbers is not hedging, because each answers a different question and neither is a product claim in disguise. The censoring rate tells a reader how much of the headline rests on tasks that actually finished rather than running out of allowance. A mean with an undisclosed censoring rate is arithmetic about a budget wearing the clothes of a capability measurement.

A policy for deciding when a number may ship

The thresholds below are a proposed policy rather than an empirical finding, and a team should set its own after looking at its baseline. The important part is deciding the rule before the run, so a disappointing censoring rate cannot be argued away afterwards.

Censoring rate	What the run may claim
under 2%	Headline score, with stop reasons linked
2% to 10%	Both scores, labelled lower bound and completed-only
above 10%	Budget adequacy only; no capability headline

Where a free tier fits in this method

A metered free tier is a sensible place to run the pilot and the estimator, because the cost of a bad projection is a smaller allowance rather than a procurement conversation. MonkeyCode is relevant here as an execution environment: the operator reports free model access and a free server option, with a free allowance advertised at ten million tokens, and those specifics should be re-checked on the project's own pages before a matrix is planned around them. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The project is described as open source, and the license and current limits are the reader's to verify rather than this article's to assume.

The method itself does not change when the envelope changes; only the numbers inside plan.json do. What matters is that the envelope is written down beside the score, so a reader can tell whether a comparison is like-for-like. An envelope that lives only in someone's memory becomes an excuse the first time two runs disagree.

Boundaries of the approach

This discipline cannot support latency or throughput claims, because metered tiers are typically rate limited and the wall clock measures the queue more than the model. It cannot support capacity planning either, since a capped run says nothing about behaviour beyond the cap. It also cannot rescue a comparison between a capped run and an uncapped one, and no amount of statistics fixes a task set that was never calibrated or a grader that was never checked.

Teams that need an absolute capability claim should use a fixed, documented envelope that someone is paying for, and should keep the ledger format anyway. Solo developers, students, and small teams deciding whether a workflow is worth adopting are the right audience for the estimator-first approach. Anyone reporting to a buyer who will read the number as a guarantee should not use a free-tier run at all.

If the pilot estimator turns out to be useful, the same harness runs unchanged against MonkeyCode's free model access and free server option, provided the current limits are checked first.