A published score is only meaningful as a distance from something, and most reports never say what that something is. Two teams can run the same model on the same task pack and publish 61% and 74%, with neither number telling you how much of the result came from the system under test and how much came from the harness, the retries, or the parser. The fix is unglamorous and rarely applied: measure the floor first, and report every score as a lift over it.
The floor is the score a task pack gives to a solver that does almost nothing. A no-op solver returns an empty answer every time; a rule-based solver applies one regex or one keyword match and stops. Neither is a model, and neither is trying to win. Their scores exist only to tell you what the pack hands out for free, which is the number any claimed improvement has to clear.
Floor runs are the first thing cut when a budget tightens, because they produce no demo and no headline. That is exactly why they belong in the method rather than in the backlog. A pack where the no-op solver already scores 40% is a pack with a label-skew or format-credit problem, and a candidate scoring 58% on it has a real lift of eighteen points, not fifty-eight.
The artifact here is small and reproducible: a holdout pack with frozen task IDs, two floor solvers, one candidate, a JSONL run log that stamps every run, and a reporting script that refuses to print a lift it cannot pair. The script is the part that matters, because it makes the floor a hard dependency instead of a good intention.
# floor_report.py - stdlib only
import json, random, sys
def load(path):
rows = []
with open(path) as fh:
for line in fh:
line = line.strip()
if line:
rows.append(json.loads(line))
return rows
def floor_by_task(rows):
"""Strongest trivial solver wins: the floor is the best free score."""
floor = {}
for r in rows:
if not r["solver"].startswith("floor:"):
continue
t = r["task_id"]
floor[t] = max(floor.get(t, float("-inf")), r["score"])
return floor
def paired_lift(rows, solver, floor):
missing = sorted({r["task_id"] for r in rows if r["solver"] == solver} - set(floor))
if missing:
raise SystemExit(f"no floor for {len(missing)} task(s), e.g. {missing[:3]}")
return [r["score"] - floor[r["task_id"]] for r in rows if r["solver"] == solver]
def bootstrap_ci(values, n=10000, seed=7):
rng = random.Random(seed)
k = len(values)
means = sorted(sum(values[rng.randrange(k)] for _ in range(k)) / k for _ in range(n))
return means[int(0.025 * n)], means[int(0.975 * n)]
if __name__ == "__main__":
rows = load(sys.argv[1])
floor = floor_by_task(rows)
for solver in sorted({r["solver"] for r in rows} - set(floor)):
lifts = paired_lift(rows, solver, floor)
lo, hi = bootstrap_ci(lifts)
mean = sum(lifts) / len(lifts)
print(f"{solver:24s} n={len(lifts):4d} lift={mean:+.3f} ci95=[{lo:+.3f},{hi:+.3f}]")
The run log needs four fields to be worth anything: task_id, solver, score, and a UTC timestamp written by the harness rather than by hand. A fifth field for the model identifier is worth adding even when the endpoint promises a stable version, because free capacity is shared capacity and version drift under a shared endpoint is a normal Tuesday.
# one pack, one window, three solvers
export STAMP=$(date -u +%Y%m%dT%H%M%SZ)
python harness/run.py --pack holdout-v3 --solver floor:noop --out runs.jsonl
export STAMP=$(date -u +%Y%m%dT%H%M%SZ)
python harness/run.py --pack holdout-v3 --solver floor:regex --out runs.jsonl
python harness/run.py --pack holdout-v3 --solver candidate --out runs.jsonl
python floor_report.py runs.jsonl
This is where free capacity changes the plan rather than the marketing. Floor sweeps are pure overhead, so they are the first runs to be dropped when each run costs money, and a protocol that depends on paying for overhead quietly loses the overhead. MonkeyCode's free model access and free server option, as described by the operator, make the floor sweep the cheap part of the study instead of the expendable one, which is the whole reason it is worth mentioning here. Disclosure: This article was prepared as part of MonkeyCode's product outreach.
Three controls keep the lift honest once the sweep is affordable. All three solvers run against the same frozen pack with the same prompt template and the same parser, because a parser change mid-sweep turns the floor into a moving target. Every run in the sweep carries a timestamp, and the report is only valid for the window it covers. Replication count is set per solver before the first run and never adjusted afterward to shrink an interval that came out wider than hoped.
The evidence tier a team may claim follows directly from what was recorded, and writing this table into the report prevents most overclaiming without any argument about statistics.
| Recorded | Permitted claim |
|---|---|
| Floor and candidate, same pack, same window | Lift with interval |
| Candidate only | Raw score, no lift |
| Floor from another day or another pack | Directional note only |
The method has limits worth stating plainly. A floor measured on Monday does not license a lift claim for a candidate run on Friday, because packs drift when tasks are added and shared endpoints drift when versions change. A no-op solver can understate the free score if the pack awards credit for refusals or for well-formed errors, so a second rule-based floor solver is worth the small extra cost. None of this corrects label errors in the pack, and none of it turns a small task count into a stable estimate.
Teams that compare scores across different packs should not adopt this as a patch, since the comparison itself is the defect. Anyone who needs one absolute number for a slide should also skip it, because the output is a paired difference with an interval and that is deliberately less quotable. For everyone else, the free lane is a reasonable place to trial the floor sweep end to end and see what the pack hands out before a single candidate run is spent.
Top comments (0)