Benchmarking LLM Agents Without the Marketing Math: Dataset, Metrics, and Controls

Last month, someone pasted a benchmark table into our team chat. Model B beat model A by 12 points on "general agent tasks," so we swapped models the same afternoon.

Two weeks later, the triage agent was mislabeling about a third of the issues it touched.

The table wasn't wrong. It was useless. It had no dataset, no metric definitions, no controls, and no decision rule. It came from a vendor blog, and a vendor blog is a marketing artifact until proven otherwise.

This post is a method, not a verdict. You'll build a small, repeatable benchmark for an LLM agent — dataset, metrics, controls, and a pre-registered decision rule — then run it on infrastructure that costs nothing. The whole harness takes about an hour to assemble.

Why most published numbers fail

A benchmark is a measurement instrument. You wouldn't trust a thermometer with no scale, yet teams routinely trust model scores with no methodology.

Three failures explain most of it:

1. No dataset. "General agent tasks" is not a dataset. You cannot re-run it, inspect it, or audit it.
2. No controls. Temperature, prompt version, model version, and run count all change results. If they are not pinned, the comparison is noise.
3. No decision rule. A number without a threshold is a conversation starter, not a decision.

The fix is boring. Build a dataset, define metrics, add controls, and write the decision rule before you run anything.

Step 1: Build a dataset from your own traces

Your dataset should look like your workload, not like a leaderboard. Sample real tasks from your logs. If you run an issue triage agent, your dataset is past issues. If you run an extraction agent, your dataset is past documents.

You need at least 30 tasks. Fewer than that, and the confidence intervals swallow any difference between models. Freeze the dataset, version it in git, and never edit it silently.

{"id": "triage_001", "input": "Fix: TypeError in parse_config when env var is missing", "expected_tool": "label_issue", "expected_args": {"label": "bug"}}
{"id": "triage_002", "input": "Docs: clarify retry behavior in README", "expected_tool": "label_issue", "expected_args": {"label": "docs"}}
{"id": "triage_003", "input": "Why is the queue stuck after a redeploy?", "expected_tool": "search_logs", "expected_args": {"query": "queue"}}

Each line is a contract: given this input, the agent should call this tool with these arguments. No free-form judging, no vibes.

Step 2: Define metrics that measure behavior

Pick one primary metric. Everything else is diagnostic. For tool-using agents, I use task success rate: the agent called the right tool with the right arguments.

def score_run(run, gold):
    if run.get("tool") != gold["tool"]:
        return {"success": 0, "tool_accuracy": 0, "early_stop": 0,
                "tokens": run.get("tokens", 0), "latency_s": run.get("latency_s", 0)}
    args_ok = run.get("args") == gold["args"]
    return {"success": int(args_ok), "tool_accuracy": 1,
            "early_stop": int(run.get("stopped", False)),
            "tokens": run.get("tokens", 0), "latency_s": run.get("latency_s", 0)}

Track four diagnostics alongside it:

• Tool-call accuracy — the right tool with wrong arguments is a different failure than a wrong tool.
• Early-stop rate — an agent that gives up is cheaper and more honest, but it is still failing.
• Cost per success — a model that wins by 3 points at 4x the cost is not a win.
• p95 latency — agents that time out in production fail in ways the success rate never shows.

Step 3: Add controls

Without controls, your benchmark measures luck. Pin everything that can be pinned:

CONTROLS = {
    "temperature": 0,
    "seed": 20260826,
    "model_version": "pinned-2026-08",
    "prompt_version": "v3",
    "runs_per_task": 5,
    "order": "randomized",
}

Run each task five times. Even at temperature zero, providers are not deterministic — load, routing, and SDK updates leak into results. Randomize task order so a provider-side outage doesn't cluster on one model. Record the model version in every output row; if the provider bumps it mid-run, your results are two benchmarks pretending to be one.

Step 4: Run it on infrastructure that costs nothing

A benchmark is only useful if you re-run it. Re-running costs money, and that is where most benchmarks die — not from bad design, but from an unpaid bill.

MonkeyCode is an open-source project, and its free model access and free server option remove that excuse. The harness below runs on a cron job, and the current free allowance of 10 million tokens is enough for many repeated evaluation runs. Quotas change, so verify the current terms before you depend on them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The free server matters for a second reason: benchmarks are flaky. A job that dies at 2 a.m. and never retries is a benchmark that lies. A tiny server with cron and a results file turns a one-off script into a monitoring loop.

0 2 * * * cd ~/agent-bench && python run_bench.py --config controls.json --out results/$(date +%F).jsonl

Every morning you get a fresh row of numbers. That's the whole trick: make the benchmark cheaper to keep than to abandon.

Step 5: Pre-register the decision rule

Write the rule before you see the output. If you decide after reading the numbers, you are not benchmarking — you are rationalizing. A pre-registered rule protects you from the most dangerous failure mode: a 2-point difference that you talk yourself into trusting.

Decision	Rule
Swap models	New model ≥ 5 points on the primary metric AND cost per success ≤ 90% of the old model
Keep current	Anything else
Re-run	Any control changed: prompt, dataset, SDK, or model version

Limitations and who should skip this

This method has real limits. Thirty tasks is a gate, not a proof — confidence intervals will be wide, and small differences between models are still indistinguishable from noise. If your dataset is public, newer models may have memorized it; keep a held-out set you never publish. And no amount of controls fixes a dataset that doesn't match your actual workload.

Skip this approach if you need regulated evaluation with human sign-off, or if you have fewer than twenty tasks — a manual checklist beats a fake benchmark. If you cannot commit to freezing the dataset, don't start. You'll get confident numbers about nothing.

The only number that survives

Model rankings change every few weeks. Your methodology is the only thing that survives contact with the next release.

So the next time someone posts a benchmark table, ask three questions: Where is the dataset? What is the primary metric? What was controlled? If the answer is a link to a blog post, you have learned nothing yet.

The harness above takes about an hour to assemble, and it runs fine on a free tier. If you build one, publish the dataset — that's the part everyone skips. If you try MonkeyCode's free tier, budget the 10 million tokens against your task count and verify the current quota first. The numbers you generate yourself are the only ones worth trusting.