Match the Human Clock Before You Quote Superhuman Coding

You get the Slack ping at 9:14. A VP pasted a thread that says coding agents already outrun most software developers. Friday is the vendor meeting, and you are the person who has to turn that thread into a hire-or-buy decision.

That thread is not a measurement. It is a mood. If you quote it in the meeting, a staff engineer will ask one question you cannot answer: under what clock, on which tasks, with which tests visible, against which humans?

This article is a protocol for that question. You will seal a holdout, freeze the snapshot, time-box both sides, and refuse any superhuman sentence the table cannot support. The numbers you emit are local results. They are not a profession-wide ranking.

The claim that keeps leaking into standups

Public agent scores and "better than developers" headlines almost never share a protocol. The agent often sees the tests. The human does not get the same tools. Retries are unlimited on one side and calendar-bound on the other.

You cannot fix that with a bigger screenshot. You fix it by matching the human protocol and by keeping a holdout the internet has never ranked.

Here is the rule you will enforce. If a junior on your team gets 45 minutes, a laptop, a failing test command, and no extra browser tab, the agent gets the same 45 minutes, the same snapshot, the same command, and the same visibility. Anything else is a demo.

What you will produce

You will leave with four artifacts, not a leaderboard row.

1. A private task pack that you do not publish.
2. A leak check against public issue titles and URLs.
3. A time-boxed runner that records hidden-test pass, time-to-green, and steer count.
4. A decision table that tells you which sentences you may say out loud.

Label every number this protocol emits as a local result. It is not market share. It is not "most developers."

Step 1 — Plant bugs in a repo the internet does not own

Do not start with famous GitHub issues. Those issues are posters. They have comments, patches, blog write-ups, and eval forks. Your holdout should be a fixture repo you control.

Create a tiny service with real tests. Then plant defects a junior could reasonably fix in under an hour. Keep gold patches off the machine the agent can read.

holdout/
  repo/                 # the snapshot both sides receive
  tasks.json            # sealed; not committed to a public remote
  denylist.txt          # public issue URLs and titles you refuse
  hidden_tests/         # pytest files the agent does not see during the run
  visible_tests/        # the command both sides may execute

A task record should look like this.

{
  "task_id": "H-04",
  "title": "invoice totals ignore refunded line items",
  "visible_command": "pytest visible_tests/test_invoice.py -q",
  "hidden_command": "pytest hidden_tests/test_invoice_refunds.py -q",
  "time_box_sec": 2700,
  "language": "python",
  "notes": "defect planted 2026-09-14; never filed on a public tracker"
}

Twelve tasks is enough to catch a dishonest workflow. It is not enough to crown a profession. Write that sentence on the README before you run anything.

Step 2 — Run a leak check before the first agent call

If a task title, stack trace, or function name already lives on a public benchmark, you do not have a holdout. You have a rerun.

Keep a denylist of URLs and phrases you will not recycle. Scan your task pack against it before every session. The script below is a tripwire, not a contamination paper.

#!/usr/bin/env python3
"""holdout_clock.py — proposed local protocol. Not a published score."""
from __future__ import annotations

import json
import re
import sys
from pathlib import Path

WORD = re.compile(r"[a-z0-9]+")


def tokens(text: str) -> set[str]:
    return {t for t in WORD.findall(text.lower()) if len(t) > 3}


def leak_hits(task: dict, denylist: list[str]) -> list[str]:
    hay = tokens(task["title"] + " " + task.get("notes", ""))
    hits = []
    for line in denylist:
        line = line.strip()
        if not line or line.startswith("#"):
            continue
        needle = tokens(line)
        if not needle:
            continue
        overlap = hay & needle
        if len(overlap) >= 4 or line.lower() in task["title"].lower():
            hits.append(line)
    return hits


def check_leak(pack_path: Path, denylist_path: Path) -> int:
    tasks = json.loads(pack_path.read_text())
    denylist = denylist_path.read_text().splitlines()
    bad = 0
    for task in tasks:
        hits = leak_hits(task, denylist)
        if hits:
            bad += 1
            print(f"LEAK {task['task_id']}: {hits[:3]}")
    print(f"checked={len(tasks)} leak_suspects={bad}")
    return 1 if bad else 0

Run it like this:

python3 holdout_clock.py check-leak

If it prints LEAK, you rewrite the task. You do not "fix" the leak by paraphrasing a well-known public issue. Plant a new bug.

This check is coarse. Four-token overlap is a stop sign, not proof that a model trained on the text. Treat a hit as do-not-use.

Step 3 — Freeze the snapshot both sides will touch

You will hand the human and the agent the same tarball. Same commit. Same lockfile. Same toolchain.

If one side gets a newer pytest plugin, you threw away the comparison. Freeze first. Argue later.

git -C holdout/repo rev-parse HEAD > holdout/SNAPSHOT.sha
python3 -m pip freeze > holdout/requirements.lock
tar -C holdout -czf holdout-snapshot.tgz repo visible_tests SNAPSHOT.sha requirements.lock
sha256sum holdout-snapshot.tgz

Store the digest next to the results. If the digest moves, the table is void. You start over.

Hidden tests stay out of the tarball. That is the point. The visible command is the only oracle both sides may use while the clock runs. You score hidden tests after the time box, on a copy the worker cannot edit.

Step 4 — Match the human clock, including the boring parts

Write the human protocol down before anyone sits down. Then copy it onto the agent runner. Do not improvise mid-session.

1. Wall clock starts at clone and stops at 45 minutes or at the first green visible suite, whichever comes first.
2. Network is off except the model API you already approved. No docs spelunking the human did not also get.
3. The agent may run visible_command as often as it wants. It may not list hidden_tests/.
4. A steer is any human edit to the prompt, the plan, or the patch after the clock starts. Count it.
5. Timeouts are failures, not "inconclusive, please retry overnight."

You now have a fairer unit than "the model eventually produced a patch." You have a session.

The runner below is a proposed local helper. It does not call a model. It only enforces the clock and the two test commands.

import subprocess
import time
from pathlib import Path


def run_time_box(task: dict, role: str, repo: Path) -> dict:
    start = time.monotonic()
    limit = int(task["time_box_sec"])
    timed_out = False
    visible_pass = False
    hidden_pass = False
    elapsed = None

    try:
        visible = subprocess.run(
            task["visible_command"],
            cwd=repo,
            shell=True,
            capture_output=True,
            text=True,
            timeout=limit,
        )
        elapsed = time.monotonic() - start
        visible_pass = visible.returncode == 0
    except subprocess.TimeoutExpired:
        timed_out = True
        elapsed = limit

    if visible_pass and elapsed is not None:
        remaining = max(1, limit - elapsed)
        hidden = subprocess.run(
            task["hidden_command"],
            cwd=repo,
            shell=True,
            capture_output=True,
            text=True,
            timeout=remaining,
        )
        hidden_pass = hidden.returncode == 0

    return {
        "task_id": task["task_id"],
        "role": role,
        "visible_pass": visible_pass,
        "hidden_pass": hidden_pass,
        "time_to_green_sec": elapsed if visible_pass else None,
        "timed_out": timed_out,
        "steer_count": None,  # fill by hand; do not invent it
    }

Fill steer_count yourself. A script cannot see you rewriting the prompt. If you leave it null and then claim the agent was unsupervised, you forged the row.

Step 5 — Record metrics marketing decks skip

Pass rate on visible tests is the demo metric. You will still log it. You will not lead with it.

Use this result schema. Fill it per task, per role.

{
  "task_id": "H-04",
  "role": "agent",
  "visible_pass": true,
  "hidden_pass": false,
  "time_to_green_sec": 1412,
  "timed_out": false,
  "steer_count": 2,
  "notes": "visible suite green; hidden refund case still red"
}

Then compute four numbers on the sealed pack only:

• Hidden pass rate. Did the patch survive tests the worker never saw?
• Time-to-green. Seconds to a green visible suite, or null on timeout.
• Steer rate. Steers per task. An unsupervised agent with a high hidden pass and zero steers is a different object than a chat window you babysat.
• Public-minus-holdout delta. If you also run a famous public set, subtract. A fat positive delta means the public set is flattering you.

None of these numbers generalize to "most software developers." They generalize to this holdout, this clock, this junior protocol. That is still enough to reject a vendor.

Step 6 — Compare with a table, not a vibe

After both roles finish, print a comparison you can paste into the Friday doc. The function below is a worksheet. It uses a mean because the code is short. If you need a median, compute a median. Do not dress sum / len as a confidence interval.

def summarize(rows: list[dict]) -> dict:
    def take(role: str) -> list[dict]:
        return [r for r in rows if r["role"] == role]

    def hidden(role: str) -> float:
        group = take(role)
        return sum(1 for r in group if r["hidden_pass"]) / len(group)

    def mean_tto(role: str) -> float:
        times = [
            r["time_to_green_sec"]
            for r in take(role)
            if r["time_to_green_sec"] is not None
        ]
        return sum(times) / len(times) if times else float("nan")

    agent = take("agent")
    return {
        "n": len(agent),
        "agent_hidden_pass": hidden("agent"),
        "human_hidden_pass": hidden("human"),
        "agent_mean_tto_sec": mean_tto("agent"),
        "human_mean_tto_sec": mean_tto("human"),
        "agent_steers": sum((r["steer_count"] or 0) for r in agent),
        "agent_timeouts": sum(1 for r in agent if r["timed_out"]),
    }

Wire the commands so a skeptic can replay the worksheet:

python3 holdout_clock.py check-leak
python3 holdout_clock.py run --role human --minutes 45
python3 holdout_clock.py run --role agent --minutes 45
python3 holdout_clock.py compare results.json

If check-leak is red, you do not run compare. You do not average a contaminated pack into a friendlier number.

When you may quote a sentence

Use this table. If a cell says no, you do not get to tighten the wording. You drop the claim.

Sentence you want to publish	Allowed when	Never allowed when
"On this sealed pack, the agent matched our junior under a 45-minute clock."	Same snapshot, same visible tests, hidden tests scored, n and steers reported	Hidden tests were in the prompt
"The agent was faster to a green visible suite on these tasks."	Time-to-green logged for both roles, timeouts counted as failures	You discarded timeouts or reran only the agent
"Public-set accuracy was higher than holdout accuracy."	You ran both sets without editing tasks after seeing misses	You tuned prompts on the holdout
"AI is better at coding than most software developers."	—	Always, from this protocol

That last row is the point. A local junior is not most developers. Twelve planted bugs are not the industry. If a vendor needs that sentence, they need a different study, with humans you actually sampled.

The dataset is the sealed pack plus the snapshot digest. The metrics are hidden pass, time-to-green, steers, and timeouts. The controls are the matched clock, the hidden tests, the denylist, and the frozen tarball. Those four controls are why the numbers are not marketing. Remove any one of them and you are back to a demo.

Where a free model and a free server fit

You still need somewhere to run the agent side without turning the eval into a credit-card subplot.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode offers free model access and a free server option. That pair is useful here because the protocol wants a stable box and a model endpoint you can call inside the time box, not a scavenger hunt across trial accounts. It does not replace the holdout. It does not turn a hidden-pass rate into a press release. If you try it, run the same holdout-snapshot.tgz you gave the human and judge the run with the table above.

Limitations, and who should not use this

This protocol will lie to you if you treat it as a championship.

It is a single language in the example. It plants bugs instead of mining production incidents. The leak check will miss paraphrases. A dozen tasks has almost no statistical power. Steer counting is only as honest as the human who logs it. A junior on your team is not a random developer, and a 45-minute clock is not a week-long feature.

Do not use this approach if you need a headline. Do not use it if you cannot recruit even one human baseline. Do not use it if your tasks already live on a public leaderboard. Do not use it to fire people. Do not use it to skip code review. A green hidden suite on a fixture repo is not a production incident closed.

What you take to Friday

You do not take a viral thread. You take a snapshot digest, a leak-check log, two role columns, and the sentences the table permits.

If the agent wins on hidden tests under the same clock, you have a local fact. If it only wins on visible tests, you have a memorizer. If it only wins after six steers, you have a pair programmer with extra latency. Name the thing you actually measured.

That is slower than quoting the internet. It is also the only version of the argument that will still make sense when the thread is gone.