An agent leaderboard becomes evidence only when the dataset slice, the metric definition, and the server session travel together. A lone pass rate can rise because the queue shortened, the fallback model changed, or an easy stratum dominated the sample. The method below treats those shifts as disqualifying events rather than as footnotes added after publication. Readers can delete every product name and still apply the same gate to ordinary local logs.
Free model access and a free server do not remove variance; they relocate it into admission control, shared queues, and silent routing. A kitchen scale on a moving truck still prints grams, yet the truck's motion is inside every reading. Benchmark authors who ignore that motion will publish a difference that belongs to the platform, not to the agent. The honest response is a pre-registered envelope for session health, not a larger headline font.
Evaluation engineers who rank coding agents need a bill of materials, much as a release engineer needs a lockfile. Without that bill, two runs that share a product name can still be different experiments in disguise. The audience here is the person who must defend a number in review, not the person who needs a demo screenshot. A demo may stay informal, while a published score cannot borrow that same looseness of setup.
The dataset in this contract is a frozen slice of tasks, not a wandering folder of prompts. Each task carries a stable identifier, a stratum label such as repair, migration, or explanation, and a content digest computed before the first run. Strata exist so that a cluster of short edits cannot outvote a handful of long migrations in the mean. The slice file is committed beside the harness, and any edit after the first scored run forces a new slice identifier.
The primary metric is a paired completion delta against a fixed baseline on the same task identifiers, not an unpaired mean across different slices. A secondary metric records timeout rate, and a third records whether the session stayed inside the health envelope. Variance is summarized with a blocked bootstrap over strata, and the publish rule demands a pre-registered minimum detectable effect. If the interval crosses zero, the harness emits no winner, because a noisy tie is not a ranking.
The control set is written into the manifest before any agent process is invoked on the slice. The baseline is a named, version-pinned runner, and the candidate sees the same task bytes, time budget, and tool permissions. A session stamp records server identity, queue delay, retry count, and a quota-state token read from the provider's live response. Runs with a missing stamp, a mid-run route change, or a quota-state outside the envelope are excluded before aggregation.
Contamination often arrives as a courtesy from the platform rather than as an obvious crash in the log. A provider may retry, shift region, or swap a route while still returning a completion that looks successful. The stamp treats that courtesy as a changed instrument, the way a chemist rejects a reading after an unlogged pipette change. Exclusion is therefore a validity rule for the instrument, not a punishment aimed at the candidate agent.
Numbers stop being marketing when a stranger can rerun the gate and obtain the same publish decision from the same logs. The harness below is a proposal with synthetic rows, and those rows are not measurements of any product. They exist so a reviewer can inspect the exclusion logic without waiting on a live network call. A real campaign replaces the fixture with captured stamps and does not reuse the toy pass flags as evidence.
#!/usr/bin/env python3
"""Session-stamp publish gate. Proposal only; rows are synthetic."""
from __future__ import annotations
import hashlib
import json
import statistics
from dataclasses import dataclass
@dataclass(frozen=True)
class TaskRow:
task_id: str
stratum: str
digest: str
baseline_pass: int
candidate_pass: int
queue_ms: int
retries: int
quota_state: str
server_id: str
route_changed: bool
ENVELOPE = {
"allowed_quota_states": {"ok", "within_budget"},
"max_queue_ms": 4000,
"max_retries": 1,
"required_server_id": "srv-lab-a",
"min_detectable_effect": 0.05,
"min_kept_tasks": 8,
}
# Synthetic fixture. Do not cite these flags as a product result.
FIXTURE = [
TaskRow("t1", "repair", "a1", 1, 1, 800, 0, "ok", "srv-lab-a", False),
TaskRow("t2", "repair", "a2", 0, 1, 900, 0, "ok", "srv-lab-a", False),
TaskRow("t3", "repair", "a3", 1, 1, 1200, 0, "ok", "srv-lab-a", False),
TaskRow("t4", "migration", "b1", 0, 0, 1500, 1, "ok", "srv-lab-a", False),
TaskRow("t5", "migration", "b2", 0, 1, 9000, 0, "ok", "srv-lab-a", False),
TaskRow("t6", "migration", "b3", 1, 1, 700, 0, "exhausted", "srv-lab-a", False),
TaskRow("t7", "explain", "c1", 1, 1, 600, 0, "ok", "srv-lab-a", False),
TaskRow("t8", "explain", "c2", 0, 1, 650, 0, "ok", "srv-other", False),
TaskRow("t9", "explain", "c3", 1, 0, 700, 0, "ok", "srv-lab-a", True),
TaskRow("t10", "repair", "a4", 0, 1, 500, 0, "within_budget", "srv-lab-a", False),
]
def slice_digest(rows: list[TaskRow]) -> str:
payload = "|".join(f"{r.task_id}:{r.digest}:{r.stratum}" for r in rows)
return hashlib.sha256(payload.encode()).hexdigest()[:16]
def keep(row: TaskRow) -> bool:
return (
row.quota_state in ENVELOPE["allowed_quota_states"]
and row.queue_ms <= ENVELOPE["max_queue_ms"]
and row.retries <= ENVELOPE["max_retries"]
and row.server_id == ENVELOPE["required_server_id"]
and not row.route_changed
)
def paired_delta(rows: list[TaskRow]) -> float:
return statistics.fmean(r.candidate_pass - r.baseline_pass for r in rows)
def blocked_means(rows: list[TaskRow]) -> dict[str, float]:
strata: dict[str, list[TaskRow]] = {}
for row in rows:
strata.setdefault(row.stratum, []).append(row)
return {name: paired_delta(group) for name, group in sorted(strata.items())}
def publish_decision(rows: list[TaskRow]) -> dict:
kept = [row for row in rows if keep(row)]
decision = {
"slice_digest": slice_digest(rows),
"n_seen": len(rows),
"n_kept": len(kept),
"dropped": [row.task_id for row in rows if not keep(row)],
"stratum_deltas": blocked_means(kept) if kept else {},
"headline": None,
"reason": "",
}
if len(kept) < ENVELOPE["min_kept_tasks"]:
decision["reason"] = "too_few_clean_tasks"
return decision
effect = paired_delta(kept)
decision["paired_delta"] = round(effect, 4)
if abs(effect) < ENVELOPE["min_detectable_effect"]:
decision["reason"] = "below_minimum_detectable_effect"
return decision
spreads = list(blocked_means(kept).values())
width = max(spreads) - min(spreads) if len(spreads) > 1 else 1.0
if effect - width <= 0 <= effect + width:
decision["reason"] = "stratum_spread_crosses_zero"
return decision
decision["headline"] = round(effect, 4)
decision["reason"] = "publish_blocked_delta_only"
return decision
if __name__ == "__main__":
print(json.dumps(publish_decision(FIXTURE), indent=2))
Running the proposal on the bundled fixture should refuse a headline, and that refusal is the expected lesson. Four synthetic rows fall outside the envelope through queue delay, an exhausted quota state, a foreign server, or a route change. Six clean rows remain, which sits below the pre-registered floor of eight, so the reason field records too_few_clean_tasks. Lowering that floor after seeing the gap would convert a written control into a marketing dial.
A reviewer reproduces the decision with one local command and then stores the JSON beside the slice file. The digest printed by the script must match the digest recorded when the slice was frozen, or the study has changed. No network call belongs in this check, because a methodology test that needs today's API cannot prove yesterday's log. Teams can wrap the command in a review bot they already trust, without adding a new leaderboard service.
python3 session_gate.py > publish_decision.json
python3 -c 'import json; d=json.load(open("publish_decision.json")); assert d["headline"] is None; print(d["reason"], d["dropped"])'
sha256sum slice.json envelope.json >> run_bill.txt
Capturing a usable session stamp is a logging problem long before it becomes a model-quality problem. Each attempt should append one JSON object with task identifier, server identifier, queue delay, retries, quota state, and route change. Missing keys are failures, and backfilling them from memory is prohibited because memory is how stale quotas enter papers. The envelope file is reviewed like a schema, and an edit to it demands the same new identifier that a slice edit demands.
Even after the gate opens, the published figure is a blocked delta on a named slice, not a universal intelligence score. Stratum means stay visible so a gain on short repairs cannot hide a loss on long migrations. Timeout rate remains beside the delta, since a faster failure is not a completion under this contract. Readers who want a single crown should look elsewhere, because this contract is built to withhold crowns.
Operators who already hold MonkeyCode free model access and a free server option can treat that environment as one candidate session. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The gate does not assume a quota size, a hardware profile, a duration, or a permanent offer, because those fields go stale. A reviewer should copy the live access terms into the session stamp and ignore any remembered allowance printed elsewhere. One practical next step is to point this same script at a private log export and withhold any ranking until exclusions are empty.
This gate is the wrong tool for exploratory chats, one-off demos, and any study that cannot freeze the task bytes. It also fails teams who need a causal claim about model quality, because a paired delta on a tiny slice is only a publish filter. Shared free servers can still correlate errors across runs, so exclusion removes some contamination without creating independence. Authors who cannot record queue delay or quota state should not publish a numeric ranking from those logs at all.
A finished report returns to that contract rather than to a product scoreboard or a remembered allowance. A dataset slice, a paired metric, and a session envelope either agree with one another, or the report stays silent. Silence is the feature that separates a lab note from a campaign graphic with extra decimals. Until the stamp, the strata, and the baseline version can be reloaded together, the printed decimals remain decoration.
Top comments (0)