Spend a Free Tier on Repeats, Not Coverage

The scarce resource on a free tier is not tokens; it is the number of independent looks you take at the same task. A pack of forty tasks run once produces a number that moves several points when a single prompt is reworded, while a pack of eight tasks run five times each produces a range that survives the rewording. The workflow below spends a free allowance on repetition, keeps the pack frozen, and reports an interval rather than a headline. It is written for teams that can afford inference but cannot yet afford confidence.

Repetition buys information that coverage cannot

Benchmark error on agent tasks is dominated by one source: the tasks themselves are not interchangeable, so the spread between a good task and a bad task is far wider than the spread between two runs of the same model. Adding tasks shrinks sampling error in the population sense, but it does not tell anyone whether the model is stable on the eight cases the product actually depends on. Repeating those eight cases does, because the per-task variance is the thing that decides whether a demo becomes a feature.

The arithmetic is unglamorous. Five repeats on eight tasks yields forty observations and eight estimates of stability, while forty tasks run once yields forty observations and zero estimates of stability. On a paid tier the first design looks wasteful, because every repeat has a price; on a free tier the second design is the wasteful one, since it spends the allowance on breadth the team has no way to interpret.

Freeze a small pack before anything else

The pack is a JSONL file, one task per line, with the answer and the grader recorded next to the prompt. The grader field keeps grading dumb and auditable, which matters more than sophistication when the sample is small. Anything that cannot be graded by exact match, substring, or a regular expression belongs in a human review sheet, not in this file.

{"id": "slug-001", "prompt": "Return the slug for 'Hello, World! App' as lowercase words joined by hyphens.", "grader": "exact", "answer": "hello-world-app"}
{"id": "json-004", "prompt": "Given {\"a\": 1, \"b\": null}, return a JSON object with null keys removed.", "grader": "regex", "answer": "\\{\\s*\"a\"\\s*:\\s*1\\s*\\}"}
{"id": "err-002", "prompt": "Explain in one sentence why a 429 response should not be retried immediately.", "grader": "contains", "answer": "rate limit"}

Hash the prompt text and store that hash with every row. When someone edits the pack mid-sweep, the hashes split the data into two populations and the mistake becomes visible in the CSV instead of hiding inside a mixed average.

A harness that runs on the standard library

The script below is a template, not a validated instrument; run it on two tasks first and inspect the rows before spending an allowance on it. It calls any OpenAI-compatible chat endpoint, records tokens and wall-clock seconds per call, and writes a row for every attempt, including failures.

#!/usr/bin/env python3
"""freetier_eval.py - repeat a frozen pack, record everything."""
import argparse, csv, hashlib, json, os, re, time, urllib.request


def load_pack(path):
    with open(path) as f:
        return [json.loads(line) for line in f if line.strip()]


def call_chat(base_url, api_key, model, prompt, temperature, seed, timeout=120):
    body = json.dumps({
        "model": model,
        "messages": [{"role": "user", "content": prompt}],
        "temperature": temperature,
        "seed": seed,
    }).encode()
    req = urllib.request.Request(
        base_url.rstrip("/") + "/chat/completions",
        data=body,
        headers={"Content-Type": "application/json",
                 "Authorization": "Bearer " + api_key},
    )
    t0 = time.monotonic()
    with urllib.request.urlopen(req, timeout=timeout) as resp:
        payload = json.loads(resp.read())
    elapsed = time.monotonic() - t0
    text = payload["choices"][0]["message"]["content"]
    usage = payload.get("usage") or {}
    return text, usage.get("total_tokens"), elapsed


def grade(task, text):
    kind, expect = task.get("grader", "contains"), task["answer"]
    if kind == "exact":
        return text.strip() == expect.strip()
    if kind == "regex":
        return re.search(expect, text) is not None
    return expect.strip().lower() in text.lower()


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--pack", required=True)
    ap.add_argument("--repeats", type=int, default=5)
    ap.add_argument("--out", default="runs.csv")
    ap.add_argument("--base-url", default=os.environ["MC_BASE_URL"])
    ap.add_argument("--model", default=os.environ["MC_MODEL"])
    args = ap.parse_args()
    key = os.environ["MC_API_KEY"]
    pack = load_pack(args.pack)
    hashes = {t["id"]: hashlib.sha256(t["prompt"].encode()).hexdigest()[:12] for t in pack}

    with open(args.out, "w", newline="") as f:
        w = csv.writer(f)
        w.writerow(["task_id", "repeat", "prompt_hash", "passed",
                    "tokens", "seconds", "status"])
        for task in pack:
            for rep in range(args.repeats):
                try:
                    text, tokens, secs = call_chat(
                        args.base_url, key, args.model,
                        task["prompt"], temperature=0.0, seed=rep)
                    w.writerow([task["id"], rep, hashes[task["id"]],
                                int(grade(task, text)), tokens,
                                round(secs, 2), "ok"])
                except Exception as exc:
                    w.writerow([task["id"], rep, hashes[task["id"]], "", "", "",
                                "error:" + type(exc).__name__])
                f.flush()


if __name__ == "__main__":
    main()

The run itself is three commands, and the environment variables are the only place provider details appear, so the same pack can be swept against a second model without editing code.

export MC_BASE_URL="https://<endpoint-from-provider>/v1"
export MC_MODEL="<model-id-listed-by-provider>"
export MC_API_KEY="<your-key>"
python freetier_eval.py --pack pack.jsonl --repeats 5 --out runs.csv

Report a range, and resample tasks rather than rows

The summary step should resample whole tasks, not individual rows, because five repeats of one task are five views of the same difficulty and treating them as independent observations shrinks the interval artificially. A clustered bootstrap over task ids costs nothing and takes a few lines.

import csv, random

def cluster_bootstrap(path, iters=2000, seed=17):
    by_task = {}
    with open(path) as f:
        for row in csv.DictReader(f):
            if row["status"] != "ok":
                continue
            by_task.setdefault(row["task_id"], []).append(float(row["passed"]))
    ids = sorted(by_task)
    rng = random.Random(seed)
    means = []
    for _ in range(iters):
        draw = [rng.choice(ids) for _ in ids]
        per_task = [sum(by_task[t]) / len(by_task[t]) for t in draw]
        means.append(sum(per_task) / len(per_task))
    means.sort()
    return means[len(means) // 2], means[int(0.025 * iters)], means[int(0.975 * iters) - 1]

With eight tasks the interval will be embarrassingly wide, and that is the honest result. The useful secondary number is the per-task swing: the difference between the best and worst task, printed next to the pooled rate. A model at 0.75 pooled with a swing from 0.2 to 1.0 is a different product decision than a model at 0.75 with a swing from 0.6 to 0.9, and only the repeat design exposes the distinction.

Spend the allowance according to the question

The allocation table below is a decision aid, not a rule; every team should measure its own interval width before trusting it.

Question being asked	Spend the free allowance on	Do not spend it on
Is the model stable on our core paths?	repeats on a frozen pack of eight to twelve tasks	new tasks
Does prompt v2 beat v1?	paired repeats, both prompts interleaved, same pack	a larger pack
What does a run cost in wall clock?	concurrency 1, fixed repeats, seconds column	token savings
Can this ship without a human check?	the worst task's rate, reported alone	the pooled mean

Controls matter as much as the design. Temperature stays at zero with the repeat index passed as a seed, and the seed is treated as best-effort because determinism is not guaranteed by every serving stack. Concurrency stays at one during timing sweeps, the model id and base URL are recorded in the article or the repo beside the CSV, and no mid-sweep model swap is allowed. Retries are recorded as rows rather than retried silently, so an error rate of 12 percent shows up as a limitation instead of disappearing into an average.

Where a free tier actually changes the arithmetic

A free allowance is what makes repetition affordable in the first place, and repetition is the only way a small pack produces an interval instead of a point. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The project is open source, and the two claims relevant here are operator-supplied: free model access for running the sweep, and a free server option for keeping the harness alive after the laptop closes. Those two claims remove the usual reason small teams run a pack once and call it a benchmark, which is that the second and third repetitions cost money.

Verify the current terms before planning a sweep around them, since allowances and server availability are exactly the kind of detail that changes between releases. Nothing in this method depends on a specific model id, quota, or machine size, and the harness records whatever the endpoint returns rather than assuming a particular response shape beyond the chat-completions convention.

Limitations, and who should not use this

Eight tasks with five repeats cannot resolve a two-point difference between two prompts, and the clustered interval will say so if it is reported honestly. Graders built from substrings reward verbosity, contamination is invisible without a holdout pack rotated out of the sweep, and free-tier throughput may make a twenty-task sweep slow enough that patience becomes the real control variable. Teams that need precision on rare failure modes, latency guarantees for a release gate, or an audit trail tied to a contractual commitment should not build that decision on a free tier at all.

The method is also a poor fit for anyone unwilling to freeze the pack, because an unfrozen pack converts every comparison into a story about which tasks were added when. Freeze it, hash it, repeat it, and publish the range with the pack attached; a number that arrives with its own interval is worth more than a headline that arrives with a screenshot. If a team wants to try the sweep without paying for inference first, the free model access and free server option described above are a reasonable place to start, provided the pack stays frozen and the interval ships with the result.