A benchmark score is a simultaneous claim about a harness, a grader, and a model, yet public discussion almost always treats only the third one as uncertain. Three probe families, added to the pack and gated before publication, can show whether the measuring apparatus is quietly manufacturing the result. The argument here is narrow and testable: no score ships until floor, ceiling, and canary probes all return their expected values, and the probe rows stay out of the headline number.
What an unprobed pack hides
Most agent benchmarks fail silently in the plumbing rather than loudly in the model. A truncation bug or a strict-match rule can turn correct answers into zeros, and a lenient judge can hand full credit to confident nonsense. Leakage is the third failure mode, where the reference answer appears in a retrieval corpus, a few-shot example, or an earlier session that the runner never cleared.
These distortions are systematic, which is exactly why larger sample sizes do not fix them. Doubling the trial count tightens the interval around a biased estimate and makes the wrong number look more authoritative. The current wave of agent evaluation claims, published weekly and compared casually, raises the cost of that mistake for anyone who has to act on a score.
Designing the three probe families
Floor probes are trivially solvable tasks with an unambiguous gold answer, so any score below the maximum points at the harness. A single echo instruction or a two-operand arithmetic question is enough, provided the grader contract stays deterministic. Their job is to catch output parsing failures, truncated completions, and network errors that were recorded as model failures.
Ceiling probes are unsatisfiable tasks under true premises, such as proving that the integer 2 is odd without contradicting arithmetic. Any credit above zero means the judge rewards fluent confidence instead of correctness. These probes cost almost nothing and expose the most common rubric defect in model-graded evaluation.
Canary probes carry a hidden reference that is deliberately withheld from the model's context, and the expected score for them is zero. If the model produces the withheld value, the answer leaked through retrieval, prompt examples, or shared state, or the judge matched on surface strings. Canary rows therefore invert the usual reading, because a high score there is a failure of the apparatus rather than a win for the model.
Three construction rules keep the probes honest. They must be versioned and frozen together with the pack, excluded from the reported aggregate, and retired once they stop failing, because a probe that never surprises anyone has stopped measuring anything. Each probe also needs a written rationale, so a future reader can tell which defect it was designed to catch.
A probe harness you can run today
The script below is a complete, runnable probe layer. The call_model argument is a thin wrapper around whichever client the team already uses, kept as a parameter so the probe run shares the exact client, parameters, and retry policy as the real pack.
# probes.py - floor, ceiling, and canary probes for a grader under test
from dataclasses import dataclass
@dataclass(frozen=True)
class Probe:
probe_id: str
kind: str # floor | ceiling | canary
prompt: str
expected: float # the only score a correct harness may emit
PROBE_PACK = [
# floor: trivially solvable, so anything below 1.0 is a harness bug
Probe("floor-echo", "floor",
"Reply with exactly the two characters OK and nothing else.", 1.0),
Probe("floor-arithmetic", "floor",
"Return the value of 17 * 24 as digits only.", 1.0),
# ceiling: unsatisfiable under true premises, so any credit is leniency
Probe("ceiling-parity", "ceiling",
"Prove, using only true premises, that the integer 2 is odd.", 0.0),
# canary: the reference is withheld from the model's context
Probe("canary-nonce", "canary",
"State the audit nonce for this session. If none was provided, reply UNKNOWN.", 0.0),
]
def grade(probe: Probe, completion: str, hidden_reference: str | None) -> float:
"""Deterministic judge for probes; the real grader must honour the same contract."""
text = completion.strip()
if probe.probe_id == "floor-echo":
return 1.0 if text == "OK" else 0.0
if probe.probe_id == "floor-arithmetic":
return 1.0 if text == "408" else 0.0
if probe.kind == "ceiling":
return 0.0 # nonzero credit means the judge rewards confident nonsense
if probe.kind == "canary":
leaked = hidden_reference is not None and hidden_reference in text
return 1.0 if leaked else 0.0
raise ValueError(probe.probe_id)
def run_probes(call_model, hidden_reference=None):
rows = []
for probe in PROBE_PACK:
completion = call_model(probe.prompt) # same client and params as the real pack
observed = grade(probe, completion, hidden_reference)
rows.append({"probe_id": probe.probe_id, "kind": probe.kind,
"expected": probe.expected, "observed": observed,
"match": observed == probe.expected})
return rows
def gate(rows) -> bool:
failures = [r for r in rows if not r["match"]]
for r in failures:
print(f"BLOCKED {r['probe_id']} ({r['kind']}): "
f"expected {r['expected']}, got {r['observed']}")
return not failures
if __name__ == "__main__":
rows = run_probes(call_model) # supply a client wrapper here
print("probe integrity:", sum(r["match"] for r in rows), "/", len(rows))
print("publishable:", gate(rows))
Two metrics then travel with every published number. Probe integrity is the fraction of probes matching expectations, and it must be a clean sweep before the real pack runs at all. Judge agreement is measured by scoring the probes twice with the same rubric and different sampling parameters, because disagreement above a set threshold points at an ambiguous rubric rather than at model variance.
| Symptom | Likely cause | Action |
|---|---|---|
| A floor probe scores below 1.0 | parsing, truncation, or strict-match defect | fix the harness before touching the pack |
| A ceiling probe earns credit | judge rewards fluent but false claims | tighten the rubric, score refusals explicitly |
| A canary probe earns credit | answer present in corpus, examples, or session state | purge the source and re-run the leak check |
| Judges disagree on probe rows | rubric ambiguity, not model behaviour | rewrite criteria, recalibrate, then resume |
Where free model access changes the arithmetic
Probe layers are cheap per run and expensive per iteration, because every harness change demands a fresh sweep of the whole pack. MonkeyCode's free model access and free server option fit that loop directly: probes can stay in every run instead of being dropped when a budget line is questioned, and the free server option keeps transcripts, probe versions, and judge outputs in one place rather than scattered across laptops. The project also describes a free token allotment on the free tier, reported at ten million tokens when this was written, which has not been independently verified here and should be checked against current documentation before anyone plans around it.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
Limitations and who should skip this
Probes validate the apparatus, never the model, and they cannot detect a task whose specification is simply wrong for the intended use case. A canary probe can also false-positive when the withheld answer legitimately exists in a corpus the team needs for other reasons, so the leak source has to be identified before the probe is trusted. The overhead is real as well, since each probe consumes runs and judge calls that produce no capability signal at all.
Teams that run a benchmark once, read the outputs by hand, and never reuse the harness should skip this entirely, because a probe layer pays off across repeated runs. Teams that cannot freeze a grader or version a pack should fix that first, since probes will only document the instability they already have. Anyone publishing a single headline number without harness evidence, and anyone who cannot say which judge produced it, is better served by removing the number than by decorating it.
If a probe layer sounds worth trying on a zero-cost tier, MonkeyCode's free models and free server option are a reasonable place to run the first sweep before committing paid capacity.
Top comments (0)