Free AI Coding Tiers Need a Protocol, Not a Press Release

A benchmark is only as honest as its protocol. Most published numbers about free AI coding tiers fail that test before the first token is spent. The reason is rarely malice; it is usually a missing control, a cherry-picked task set, or a cost column that nobody kept. This article defines a minimal benchmark protocol with a frozen dataset, three behavioral metrics, and five documented controls. The protocol is deliberately small because a reproducible benchmark beats an impressive one.

The typical vendor post reports a single accuracy figure on tasks chosen after the fact. That is like timing a car on a downhill road with a tailwind. Best-of-N sampling inflates the result further because the model runs each task twenty times and the author keeps the one attempt that passed. Cost is excluded entirely, which matters enormously when the tier is free because free changes the economics of every decision that follows. A benchmark becomes trustworthy only when the dataset is frozen, the metrics are behavioral, and the controls are documented next to the results.

The dataset is the part that most people rush, and it determines whether the result means anything at all. A credible set contains twenty to thirty tasks drawn from real repositories, each with a failing test and a prompt that does not appear in any vendor's marketing material. Popular tasks from public leaderboards should be avoided because they are almost certainly in training data, which turns the exercise into a memorization check. Each task should fit in one context window but require real reasoning, such as implementing a missing function or fixing a subtle off-by-one error.

Three metrics capture the behavior that actually matters: pass rate, token cost per solved task, and wall-clock time per task. Pass rate is pass@1 with temperature zero, meaning the model gets one attempt and the hidden test decides the outcome. Token cost is measured from the prompt and completion counts, then divided by the number of solved tasks so cheap and dumb competes against expensive and smart on equal footing. Wall-clock time matters for free tiers because throttling can make a good model unusable during peak hours, and the median across runs is the right summary statistic.

Five controls separate a protocol from a press release, and each one closes a specific hole that would otherwise invalidate the result. Temperature is fixed at zero, the system prompt is identical across models, and the context window is the same size for every run. Each task is executed five times so variance is reported instead of hidden, and the entire suite runs at the same time of day to control for quota throttling. The final control is the most important one: the harness is published with the results, because an unreproducible benchmark is a story, not a measurement.

Here is the harness, kept deliberately small so that reviewing it takes less time than running it. The endpoint and key come from environment variables, which means the same script works against any OpenAI-compatible API. The three helper functions are provider-specific stubs, because the point of the protocol is the structure, not the SDK glue.

# bench_free.py — a minimal honest benchmark for free AI coding tiers
import json
import statistics
import time
from pathlib import Path

TASKS = Path("tasks.json")
TEMPERATURE = 0.0
RUNS = 5

def load_tasks():
    return json.loads(TASKS.read_text())

def run_task(model, task):
    prompt = f"{task['context']}\n\nImplement the missing function:\n{task['prompt']}"
    start = time.monotonic()
    patch = model.complete(prompt, temperature=TEMPERATURE)
    elapsed = time.monotonic() - start
    passed = run_hidden_test(patch, task["test"])
    return {
        "task": task["id"],
        "passed": passed,
        "seconds": elapsed,
        "prompt_tokens": count_tokens(prompt),
        "completion_tokens": count_tokens(patch),
    }

def summarize(results):
    rows = []
    for task_id in {r["task"] for r in results}:
        runs = [r for r in results if r["task"] == task_id]
        rows.append({
            "task": task_id,
            "pass_rate": statistics.mean(r["passed"] for r in runs),
            "median_seconds": statistics.median(r["seconds"] for r in runs),
            "median_tokens": statistics.median(
                r["prompt_tokens"] + r["completion_tokens"] for r in runs),
        })
    return rows

def main():
    model = load_model_from_env()
    results = []
    for task in load_tasks():
        for _ in range(RUNS):
            results.append(run_task(model, task))
    print(json.dumps(summarize(results), indent=2))

if __name__ == "__main__":
    main()

The tasks.json file is the frozen dataset, and it should be committed to version control before the first run. That way nobody can edit the tasks after seeing the results, which is the most common form of accidental cheating. A minimal entry looks like this, with the context, the prompt, and the hidden test that decides pass or fail.

{
  "id": "repo_007_off_by_one",
  "context": "def last_index(items, target):\n    # returns the last index where target appears, or -1",
  "prompt": "Implement last_index so it handles empty lists correctly.",
  "test": "assert last_index([1, 2, 3, 2], 2) == 3\nassert last_index([], 5) == -1"
}

Then the run is a single command, and the output is a JSON table that can be committed next to the code.

export MODEL_ENDPOINT="https://your-free-tier-endpoint/v1"
export MODEL_KEY="$YOUR_KEY"
python bench_free.py > results.json

A free tier changes the benchmark in one important way: the token allowance becomes part of the measured cost, not a discount applied after the fact. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project that offers free model access and a free server option, and as of this writing it advertises a ten-million-token allowance that lets the harness above run without a credit card. The free server option matters for benchmarking because it removes your own infrastructure from the measurement, so the numbers reflect the model and not a noisy laptop. Running the harness against that endpoint is the fastest way to see the protocol in action, and the resulting JSON will tell you more than any screenshot. Token allowances and server limits change over time, so the honest move is to read the repository before trusting any number quoted in a blog post.

This protocol has real limits, and teams that ignore them will over-trust the output. A twenty-task dataset cannot detect regressions in long-horizon agentic workflows, where a model fails after fifty steps instead of five. The harness measures single-shot code generation, so it says nothing about interactive debugging, cross-file refactoring, or how a model behaves when the reviewer writes the tests. Teams that need production-grade evidence should run a human-reviewed subset against a baseline model they already trust, rather than treating any automated number as the final word.

The point of a benchmark is not to crown a winner but to make the next decision cheaper, and a frozen dataset with published controls does exactly that. Anyone evaluating a free coding tier can run this protocol in an afternoon and keep the results for the next release. The numbers will age, the models will change, and the protocol will still be there, which is the only property that makes a benchmark worth publishing.