Keep the Denominator Frozen: A Checker for Agent Pass Rates

You closed the laptop on a green terminal. Four tasks, two passes, a free server still spinning. In the team channel the number had already become "50% solved."

Nobody attached the task list. Nobody said what "solved" ignored.

This is a procedure for freezing that meaning before the run, then refusing to print a rate when the logs drift. The artifact is a small checker you can run locally. It does not rank vendors.

The failure you are actually looking at

A percentage looks precise. It is only as stable as the denominator.

Change one task ID after you peek, and the rate moves. Drop a timeout, and it moves again. Call an empty patch a pass because the agent wrote a confident summary, and you have left measurement entirely.

You want the opposite habit. Decide the slice, the metrics, the control, and the access path. Then generate the number, or generate an error.

Step 1: Commit the contract while run_id is still null

Put scoring_contract.json in the eval repo. JSON keeps the checker free of extra dependencies. YAML is fine later if you already pin a parser.

{
  "contract_version": 1,
  "run_id": null,
  "locked_at": null,
  "task_ids": [
    "t_03_rename_helper",
    "t_11_fix_off_by_one",
    "t_18_add_missing_test",
    "t_22_reject_network_call"
  ],
  "metrics": {
    "primary": "task_pass",
    "task_pass": "tests_green AND patch_applies_clean AND no_forbidden_path",
    "abstain": "exit_code_0 AND patch_empty",
    "timeout": "wall_seconds > 180",
    "post_lock_exclusion": "forbidden"
  },
  "controls": {
    "narration_probe": "summary only; a non-empty patch fails the probe"
  },
  "access_class": {
    "host": "unspecified",
    "model_access": "unspecified",
    "quota_assumed": false,
    "permanence_assumed": false
  }
}

This file is a proposal you fill in, not a published score. Four tasks are a smoke slice. They can expose a sloppy denominator. They cannot support a model comparison.

Field notes, in the order you should fill them:

1. task_ids is the denominator. A folder glob is not allowed at score time.
2. task_pass is a conjunction. Any false clause fails the task, including a dirty apply or a forbidden path.
3. narration_probe is a control condition. It is not a second leaderboard and it does not enter the numerator.
4. access_class records where the run happened. Treat it as a condition, not as a trophy.

Step 2: Record the host you actually used

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

You can point access_class.host at MonkeyCode's free server option and model_access at its free model access. Those two availability claims are operator-supplied for this draft. They are the reason a locked four-task slice is cheap to execute.

Do not write a token quota, a hardware name, or a duration into the contract unless a current primary page states them and you store the URL plus the date you read it. This article does not. Free paths can be queued or withdrawn. Set quota_assumed and permanence_assumed to false and leave them there.

If a teammate later reruns the same IDs on another host, mint a new run_id. Averaging across access classes hides the variable you forgot to name.

The product is a place to run the protocol. It is not the conclusion of the protocol. Delete the name and the checker still tells you when a rate is unfit to quote.

Step 3: Lock, then refuse an unlocked rate

Save the checker as refuse_unlocked_rate.py. The sample below is unexecuted teaching code. Run it on your own fixture before you trust the exit codes.

#!/usr/bin/env python3
"""Emit a pass rate only for a locked scoring contract."""

import json
import sys
from datetime import datetime, timezone
from pathlib import Path

REQUIRED = ("task_ids", "metrics", "controls", "access_class")

def load(path):
    return json.loads(Path(path).read_text(encoding="utf-8"))

def dump(path, obj):
    Path(path).write_text(json.dumps(obj, indent=2) + "\n", encoding="utf-8")

def lock(path, run_id):
    contract = load(path)
    if contract.get("locked_at"):
        sys.exit("already locked; start a new contract file")
    missing = [k for k in REQUIRED if not contract.get(k)]
    if missing:
        sys.exit(f"refusing to lock incomplete contract: {missing}")
    access = contract["access_class"]
    if access.get("quota_assumed") or access.get("permanence_assumed"):
        sys.exit("refusing to lock assumed quota or permanence")
    if access.get("host") in (None, "", "unspecified"):
        sys.exit("set access_class.host before lock")
    contract["run_id"] = run_id
    contract["locked_at"] = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
    dump(path, contract)
    print(f"locked {run_id} at {contract['locked_at']}")

def score(contract_path, events_path):
    contract = load(contract_path)
    events = load(events_path)
    if not contract.get("locked_at") or not contract.get("run_id"):
        sys.exit("unlocked contract; no rate emitted")
    access = contract["access_class"]
    if access.get("quota_assumed") or access.get("permanence_assumed"):
        sys.exit("assumed quota or permanence; no rate emitted")
    allowed = list(contract["task_ids"])
    if len(allowed) != len(set(allowed)):
        sys.exit("duplicate task_ids")
    by_id = {}
    for ev in events:
        tid = ev["task_id"]
        if tid not in allowed:
            sys.exit(f"event outside locked slice: {tid}")
        if ev.get("excluded_after_lock"):
            sys.exit(f"post-lock exclusion starts a new study: {tid}")
        if "task_pass" not in ev:
            sys.exit(f"missing task_pass: {tid}")
        by_id[tid] = ev
    missing = [tid for tid in allowed if tid not in by_id]
    if missing:
        sys.exit(f"denominator drift, missing {missing}")
    passed = 0
    for tid in allowed:
        ev = by_id[tid]
        if ev.get("abstain") and ev.get("task_pass"):
            sys.exit(f"abstain cannot pass: {tid}")
        if ev.get("task_pass") is True:
            passed += 1
    print(json.dumps({
        "run_id": contract["run_id"],
        "locked_at": contract["locked_at"],
        "n": len(allowed),
        "task_pass_count": passed,
        "task_pass_rate": passed / len(allowed),
        "access_class": access,
        "note": "valid only for this slice, metric, and access class"
    }, indent=2))

if __name__ == "__main__":
    cmd = sys.argv[1]
    if cmd == "lock":
        lock(sys.argv[2], sys.argv[3])
    elif cmd == "score":
        score(sys.argv[2], sys.argv[3])
    else:
        sys.exit("usage: lock|score")

Fill the host with a label you can defend, then lock. Scoring before that step should fail.

python3 - <<'PY'
import json
p = "scoring_contract.json"
with open(p, encoding="utf-8") as f:
    c = json.load(f)
c["access_class"]["host"] = "operator-supplied free server option"
c["access_class"]["model_access"] = "operator-supplied free model access"
with open(p, "w", encoding="utf-8") as f:
    json.dump(c, f, indent=2)
    f.write("\n")
PY
python3 refuse_unlocked_rate.py lock scoring_contract.json run_2026_10_12_a
python3 refuse_unlocked_rate.py score scoring_contract.json events.json

The script does not fetch a quota. It only checks that you did not pretend to have one.

events.json must contain every locked ID, including failures:

[
  {"task_id": "t_03_rename_helper", "task_pass": true, "abstain": false, "wall_seconds": 40},
  {"task_id": "t_11_fix_off_by_one", "task_pass": false, "abstain": false, "wall_seconds": 55},
  {"task_id": "t_18_add_missing_test", "task_pass": false, "abstain": false, "wall_seconds": 200},
  {"task_id": "t_22_reject_network_call", "task_pass": true, "abstain": false, "wall_seconds": 30}
]

On this fixture the checker would print task_pass_count 2 and n 4. That printout is an example of the schema. It is not a measurement of any agent. Replace the booleans with values your harness computed from tests.

Step 4: Make exclusions and probes impossible to clean up

Run the narration probe on the same IDs before you trust task_pass. The prompt asks for a summary and forbids a patch. A non-empty diff fails the probe.

Keep that result beside the rate. Do not fold it into the numerator.

Timeouts remain in n. An abstain remains in n and cannot set task_pass. A test you discover is flaky after locked_at does not get an excluded_after_lock flag on this run. Copy the contract, explain the flake, and score a new run_id.

If your harness has no hidden tests, write that inside metrics. Silence is how a public sample becomes a private-sounding claim.

Why the printed rate is still not marketing

A marketing number is allowed to move after the audience reacts. A measurement is not.

You publish the contract hash with the rate. You publish the misses. You publish the access class as a constraint. You do not publish a comparative adjective.

"Better" needs a paired design you did not run: same IDs, same budget policy, same tests, a second access class, and a rule for ties written in advance. Until that second contract exists, the honest sentence is local. On this locked slice, under this access class, this many tasks met task_pass.

That sentence will not travel as a billboard. Good.

Evidence sentence	Slogan it must not become
"2 of 4 task_pass on this locked run_id."	"50% accuracy."
"Narration probe emitted no patch on this slice."	"The harness is fair."
"Access class recorded; terms read that day."	"Free hosting means the score will hold."
"Flake deferred to a new run_id."	"We removed the bad task and improved."

The left column is dull. Dull is the feature. A marketing line needs the right column, which this checker is built to withhold.

Step 5: Store a bundle a stranger can rerun

A rate without a bundle is a rumor. After the checker exits 0, write a directory that contains only boring files.

runs/run_2026_10_12_a/
  scoring_contract.json
  events.json
  narration_probe.json
  commands.txt
  README.md

commands.txt lists the test invocation, not a paraphrase.

sha256sum scoring_contract.json events.json > runs/run_2026_10_12_a/SHA256SUMS
python3 refuse_unlocked_rate.py score scoring_contract.json events.json | tee runs/run_2026_10_12_a/rate.json

The README gets five lines: slice purpose, access class, date you read the host terms, probe result, and the sentence "not a vendor ranking." If you cannot write that last line without hedging, you are not done.

Who should not use the approach

Do not use a four-task lock to choose a vendor, set a price, or announce a winner. The sample is too small, and this draft supplies no confidence interval you can hide behind.

Do not use it if you cannot keep the patch, the test command, and the contract file. A chat scroll cannot be re-scored.

Do not use it when task text is copied from a popular public repo and you have no note about possible overlap with model training data. Say "contamination unknown" next to the rate, or do not publish the rate.

Skip the free-server path if your question depends on a reserved machine, a fixed region, or a paid throughput guarantee. Those properties are outside the operator-supplied claims used here.

Limits of the checker

The script trusts task_pass. You still have to fill that boolean from a test command, not from model prose. A lying harness makes a locked lie.

Clock skew, queue delay, and partial outages on a free host belong in the event log as timeouts or as a failed run, not as deleted rows. This article states no latency budget and no uptime figure.

The sample events are unexecuted illustrations. Do not cite them. Do not add a model name to the JSON to make a screenshot look finished.

Ship the smaller sentence

Lock four tasks you can explain without a diagram. Run the probe. Run one agent pass. Paste the checker JSON, or paste the error.

If that log is something your team will rerun, the operator-supplied free model access and free server option are a reasonable place to do the next slice. Read the live terms on the day you lock the contract, store the date in your notes, and keep the claim as narrow as the run_id.