Calibrate the Task Before the Model: A Spec-Only Floor for Agent Benchmarks

You publish a table. Model A sits at 71% on your patch suite, model B at 64%. Someone asks a simple question — how many of those tasks are solvable from the issue text alone? — and the thread goes quiet.

That silence is the bug. Seven points of separation means nothing if four of your tasks never needed the repository in the first place. Before you compare endpoints, you have to know how much of your dataset a prompt alone can already solve.

This post is a working method for that: a spec-only floor, computed per task, cheap enough to run on a free tier, and boring enough to publish alongside your scores.

The problem is task-side, not model-side

Most agent benchmarks are built by collecting real issues and real patches. That is good for realism and terrible for calibration. A task titled "rename getUserData to fetchUser" is a find-and-replace. A task titled "fix flaky integration test" is a research project. Both land in the same average.

Worse, the easy end of the set is not random noise — it correlates with how much context leaked into the issue body. Stack traces, file paths, and pasted diffs all shrink the search space. So your pass rate partly measures how well you wrote your tickets.

The fix is not a harder dataset. It is a measurement of the dataset itself, taken with the same harness you use for models.

Step 1 — Define two passes over the same task

Every task gets run twice, with only one variable changed: repository access.

1. Spec pass — the agent receives the issue text, the test command, and the language/runtime version. No checkout, no file listing, no grep.
2. Full pass — identical prompt, identical budget, plus a read-only checkout of the repo at the recorded parent commit.

Everything else is pinned: same model endpoint, same temperature, same max steps, same wall-clock cap, same judge.

If the spec pass succeeds, the task did not require retrieval. That is a dataset fact, not a model opinion.

Step 2 — Record the floor in the dataset, not in a spreadsheet

Put the field in the task record so it travels with the data. A task line looks like this:

{"id":"t-0142","repo":"svc-billing","base_commit":"9f3c1ab","issue":"...","test_cmd":"pytest -q tests/test_invoice.py","floor_pass":0,"floor_n":3}

Then the scoring script reads it instead of recomputing it. Here is the harness sketch I use for the report. It is unexecuted here — wire run_task() to your own runner and keep the comparison logic untouched.

# floor_report.py — SKETCH, not run as written
import json, statistics
from collections import defaultdict

def load(path):
    with open(path) as f:
        return [json.loads(line) for line in f if line.strip()]

def run_task(task, mode, model, seed):
    """mode='spec' -> issue only. mode='full' -> issue + checkout."""
    # delegate to your existing sandboxed runner; return True/False
    raise NotImplementedError

def measure(tasks, mode, model, repeats=3):
    results = defaultdict(list)
    for t in tasks:
        for seed in range(repeats):
            results[t["id"]].append(run_task(t, mode, model, seed))
    return {k: sum(v) / len(v) for k, v in results.items()}

def report(tasks, model, repeats=3):
    spec = measure(tasks, "spec", model, repeats)
    full = measure(tasks, "full", model, repeats)
    rows = []
    for t in tasks:
        f, p = spec[t["id"]], full[t["id"]]
        rows.append((t["id"], f, p, p - f))
    keep = [r for r in rows if r[1] == 0]
    print(f"tasks={len(rows)} floor_clean={len(keep)}")
    if keep:
        print(f"mean_delta_over_floor={statistics.mean(r[3] for r in keep):.3f}")
    return rows

if __name__ == "__main__":
    report(load("tasks.jsonl"), model="<your-endpoint>")

The output you care about is floor_clean and the delta. Raw pass rate is not the headline.

Step 3 — Run the floor pass first, because it is the cheap one

Order matters. Spec passes use a fraction of the tokens and none of the sandbox disk, so they finish before your full runs start. That is exactly the profile where a free tier is useful rather than embarrassing.

MonkeyCode is the project I have been using for this pass: the operator advertises free model access and a free server option, and the allowance quoted to me for the free tier was 10 million tokens. Treat that number as a snapshot, not a contract — quotas on free tiers move, so check the page yourself before you size a run.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I run the floor with concurrency 2 and a hard 90-second cap per task:

python floor_report.py --tasks tasks.jsonl --concurrency 2 --max-seconds 90

Ten tasks is enough to find your first contaminated item. You do not need the whole suite to learn that your bugs are well documented.

Step 4 — Score the delta, and say what each metric means

Report three numbers, not one. Collapsing them is where tables start to resemble marketing.

Metric	Definition	What it answers
F	mean spec-pass rate over n repeats	how much of the task is prompt-solvable
P	mean full-pass rate over n repeats	the number you used to publish alone
D = P − F	per-task delta, averaged over floor-clean tasks	value added by repository access

D is the honest comparison axis. Report F next to it, and report the repeat spread — the min and max across seeds — because a delta with a four-point spread across three runs is a coin arranging itself.

Step 5 — Pin the controls, or the floor drifts too

Five controls, all boring, all necessary:

1. Freeze the checkout at base_commit; no git pull between runs.
2. Same runtime image for both passes. The spec pass must not get a fresher toolchain.
3. Same judge and the same pinned rubric version for both passes, including the spec pass. A lenient judge inflates F and deflates D.
4. Same step and time budget. Giving the full pass more steps is a second variable you did not mean to add.
5. Log the prompt hash. If a task prompt changes mid-run, the floor is stale.

Step 6 — Act on the floor with a decision table

Do not delete tasks on instinct. Classify them.

Spec pass over 3 tries	Reading	Action
0/3	needs the repo	keep in the headline set
1–2/3	prompt hints are half the answer	keep, but report in a separate stratum
3/3	solved without retrieval	move to a smoke suite, or drop

The middle row is the one people delete by accident. Those tasks are not broken — they are just easier than the average they were hiding inside.

Limitations, and who should skip this

The spec-pass floor conflates two things: genuine prompt leakage and genuinely easy tasks. It also says nothing about whether a task is important, only whether it is self-contained. With n=3 repeats the estimate is coarse, and a task sitting at one success out of three is a coin flip you should not over-read.

Skip this method if you need absolute leaderboard numbers comparable to someone else's suite, if your tasks depend on private infrastructure you cannot sandbox twice, or if you already have enough budget to run a full factorial design — in that case the floor is a redundancy, not a discovery.

If you do try it, start with ten tasks from your existing suite and run only the spec pass. The free tier on MonkeyCode covers that comfortably, and the list of tasks your prompt alone can solve is usually more useful than the scoreboard it was hiding in.

Comments

[Alex Shev]

A spec-only floor is a strong control because it tests whether the task itself has a stable notion of success. I’d version the spec, fixtures, grader, and allowed tools together, then report how often competent humans disagree with the score. If the benchmark cannot survive that calibration, model rankings are measuring evaluator noise.