Measure the Public-Hidden Gap Before You Cite a Coding-Agent Score

You are one comment away from putting a pass rate in a team channel. The agent repaired 16 of 24 public issues. That is 67%, and it looks like a result.

It is not a result yet. Those issues were public. The hidden tasks are still unscored. A canary that should have failed has not been run. Until you measure that gap, the percent is a story with a denominator.

This is a contamination audit, not a leaderboard recipe. You will split the dataset, freeze the tool surface, and compute a public-hidden gap plus an inverted canary rate. If the gap is wide or a canary passes, you publish the failure of the protocol. You do not publish a single score.

Start from the failure you are trying to prevent

The failure is memorization dressed up as repair. A model can emit a known patch because the issue text was in pretraining, or because your prompt included a hint, or because the test file was world-readable in the sandbox.

You cannot prove a negative with one run. You can make the easy cheating paths visible. Do that with slices, not with a longer introduction in the README.

Step 1: Build three slices and write them into the task file

Keep the set small. Twenty-four public tasks, twenty-four hidden tasks, and eight canaries is enough to embarrass a sloppy protocol. It is not enough to rank vendors.

1. Public slice. Issue text and a gold patch already exist on the public web. Every pass here is guilty of possible memorization until the hidden slice agrees.
2. Hidden slice. Tasks you wrote, or tasks you withheld. Same languages and similar edit size as the public slice. The gold patch is not in the prompt, not in a gist, and not in the repository the agent can read.
3. Canary slice. Tasks that must not pass. They exist to catch a harness that paints stubs green.

Drop any row you cannot label. An unlabeled row will be averaged later by someone in a hurry, and that average is how a marketing number starts.

Match difficulty on purpose. If hidden tasks are one-function toys and public tasks touch five files, the gap measures your sampling. Write a one-line difficulty note before you run anything.

# dataset_card.yaml — template, not a released benchmark
dataset_id: contamination-audit-2026-09-25
commit: "replace-with-your-git-sha"
slices: {public: 24, hidden: 24, canary: 8}
languages: [python, go]
difficulty_note: "one failing test, at most two edited files"
attempt_cap: 1
temperature: 0
network_on_hidden: false
gold_patch_in_prompt: false

Hash the card with the prompt so a later edit cannot silently retune the set.

sha256sum prompt_template.txt dataset_card.yaml
git rev-parse HEAD

Store those digests in the outcome file. A score without them is not replayable.

Step 2: Construct canaries that an honest run fails

A canary is not a trick question for the model. It is a trap for your grader.

1. Missing path. The prompt names billing/tax.py. The repo has no such file, and success requires a real diff. A pass means the harness accepted a no-op, or invented a file and still went green.
2. Inconsistent oracle. The prompt forbids changing exception types. The checked-in test demands a new exception. No correct patch exists. A pass means tests were swapped or not run.
3. Decoy comment. A non-imported file contains a comment with a plausible fix. The agent should not need that file. If passes show up only when that file is on the tool path, you have a context leak, not a repair.

Eight canaries, mixed across those three shapes, is a start. Record passed: true only if the harness marked the task successful. On this slice, true is the bad bit.

Step 3: Freeze the tool surface before the first attempt

Two agents with the same weights and different tools are different systems. File read, shell, network, a test runner, and a hosted sandbox all move the pass rate.

Write the surface down first.

1. Access label, copied from the provider docs on the day you run, not from memory.
2. Where tests executed: local container, CI image, or a hosted server.
3. Network policy: off, allowlist, or open. Hidden tasks should be off.
4. Tool list. If shell is present, say so in the card.
5. Attempt cap, temperature, and the context limit you actually set.
6. Prompt hash and dataset commit from the commands above.

Do not average a laptop run with a hosted-server run. Report them as separate surfaces. A free server you do not control can change image, network policy, or availability between weeks. If you cannot snapshot it, those two weeks are not the same system.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. If your run uses MonkeyCode's free model access or its free server option, those are tool-surface values, not evidence that the score improved. This article does not name models, token quotas, hardware, or how long that access lasts. Those details change, and none were verified against a primary document here. Read the current notes, then log only what your process actually saw.

Step 4: Compute metrics that are allowed to blank the headline

You need slice rates, a gap, a canary fail count, and a lower bound. You do not need a composite score.

Metric	How you compute it	When it blocks a headline
Slice pass rate	Passes divided by tasks in that slice	Mixed tool surfaces in one table
Public-hidden gap	Public rate minus hidden rate	Gap above 0.25 with at least 20 tasks in each slice
Canary fail rate	Failed canaries divided by canary count	Any canary pass, or zero canaries
Wilson lower bound	95% lower bound on the hidden pass rate	Prefer this bound over a bare percent when you describe the hidden slice

The 0.25 cutoff is a reporting policy, and it is intentionally wide. A smaller gap can still be leakage, noise, or a difficulty mismatch you failed to write down. The cutoff exists to stop a one-number slide, not to certify purity.

Here is a fictional hand count, so you can see the rule without borrowing a vendor chart. Public 16/24 is about 0.67. Hidden 7/24 is about 0.29. The gap is about 0.38, and canaries failed 8/8, which is what you want.

The gap still blocks a headline. A Wilson 95% lower bound on 7/24 lands near 0.15. You may describe the hidden slice as weak and uncertain. You may not say the agent repairs two thirds of issues.

Step 5: Gate the outcome file

The following script is an unexecuted example. It does not call a model, and it does not contain a measured score. Point it at JSONL you collected yourself.

{"id":"h-014","slice":"hidden","passed":false,"tool_surface":"local-container","prompt_sha256":"abc"}

#!/usr/bin/env python3
"""Contamination gate. Unexecuted example, not a model result."""
import json
import math
import sys
from collections import defaultdict

GAP_LIMIT = 0.25
MIN_SLICE = 20

def wilson_lower(passed: int, n: int, z: float = 1.96) -> float:
    if n == 0:
        return 0.0
    p = passed / n
    denom = 1 + z**2 / n
    center = p + z**2 / (2 * n)
    margin = z * math.sqrt((p * (1 - p) + z**2 / (4 * n)) / n)
    return max(0.0, (center - margin) / denom)

def load(path: str):
    rows = []
    with open(path) as fh:
        for line in fh:
            if line.strip():
                rows.append(json.loads(line))
    return rows

def rate(rows):
    n = len(rows)
    passed = sum(1 for r in rows if r["passed"])
    return passed, n, (passed / n if n else 0.0)

def main(path: str) -> int:
    rows = load(path)
    surfaces = {r.get("tool_surface") for r in rows}
    hashes = {r.get("prompt_sha256") for r in rows}
    by_slice = defaultdict(list)
    for row in rows:
        by_slice[row["slice"]].append(row)

    blockers = []
    if len(surfaces) != 1:
        blockers.append("mixed tool_surface; split the report")
    if len(hashes) != 1:
        blockers.append("mixed prompt hash; freeze the template")

    summary = {}
    for name in ("public", "hidden", "canary"):
        passed, n, p = rate(by_slice.get(name, []))
        summary[name] = {"passed": passed, "n": n, "rate": round(p, 4)}

    public, hidden = summary["public"], summary["hidden"]
    if public["n"] < MIN_SLICE or hidden["n"] < MIN_SLICE:
        blockers.append("slice smaller than 20; no headline")
    gap = public["rate"] - hidden["rate"]
    if public["n"] >= MIN_SLICE and hidden["n"] >= MIN_SLICE and gap > GAP_LIMIT:
        blockers.append(f"public-hidden gap {gap:.2f} exceeds {GAP_LIMIT}")
    if summary["canary"]["n"] == 0:
        blockers.append("no canaries")
    elif summary["canary"]["passed"]:
        blockers.append(f"{summary['canary']['passed']} canary passes")

    report = {
        "slices": summary,
        "public_hidden_gap": round(gap, 4),
        "hidden_wilson_lower_95": round(
            wilson_lower(hidden["passed"], hidden["n"]), 4
        ),
        "headline_blocked": bool(blockers),
        "blockers": blockers,
    }
    json.dump(report, sys.stdout, indent=2)
    print()
    return 2 if blockers else 0

if __name__ == "__main__":
    sys.exit(main(sys.argv[1]))

python3 contamination_gate.py outcomes.jsonl
echo "exit=$?"

Exit 2 means the headline stays out of the channel. Exit 0 means the contamination gate cleared, not that the model is good. A hidden rate of 0.10 can clear it. That is a weak repair rate you are allowed to describe, and it is not a launch claim.

Step 6: Decide what you are willing to say

Read the JSON. Do not bargain with it in the pull request text.

1. Any blocker. Post the blockers, the slice counts, and the tool-surface card. No single percent.
2. Gate clear, hidden lower bound under 0.30. Say the protocol held and the repair rate is low. Print the bound next to the rate.
3. Gate clear, but your difficulty notes already predicted a gap. Report slices separately. Do not average them away.
4. Canary pass. Stop. Fix the harness before you change models, prompts, or servers.

A number is marketing when it hides the slice that looks worse, mixes sandboxes, or counts a canary success as skill. Arithmetic does not rescue that slide.

Limitations

This audit does not prove the hidden set is unseen. A private task can still rhyme with a famous bug. A similarity check of your hidden prompts against public issues is a useful extra control, and this script does not do it.

The gate ignores cost, latency, and token use. Track those on a separate card. Do not average pass rates across unequal budgets and call the blend a quality gain.

Wilson intervals assume independent trials. Retries, a shared repo cache, and one flaky test break that assumption. If you raise the attempt cap, change the dataset card in the same commit, or the bound is decoration.

The script in this article has not been run on a live agent. No model score is reported here.

Who should skip it

Skip this if you need a comparison slide today. A hidden slice takes longer to write than a prompt tweak, and that delay is the point.

Skip it for exploit development, malware, or other offensive tasks. A pass rate is the wrong artifact there, and this harness will not make that work safer.

Skip it if gold patches must live in the prompt for the product you are testing. Without a withheld oracle, you are scoring autocomplete against the answer key.

Also skip it when the deliverable is a demo. A demo can be honest without wearing a benchmark costume.

What you put in the note instead

Put the dataset commit, the two hashes, the tool-surface card, three slice rates, the gap, the canary fail count, and the hidden Wilson lower bound. If a free server or free model access was in the path, name that surface and link the access notes you read that day. Do not repeat a quota from memory.

That note is harder to skim than 67%. It is also the version a teammate can rerun after you rotate the hidden set.

Before you freeze the commit, ask someone who did not write the tasks to relabel ten hidden items for difficulty. Disagreement at that step is cheaper than a correction after the number has traveled.