Plant Bugs Your Agent Must Find: A Self-Test for Coding-Agent Benchmarks

The one question that invalidates a benchmark

Your suite says the agent fixed 91% of the bugs. Someone in review asks: has this harness ever returned a negative result on a task where you already knew the answer? You scroll the logs. It has never failed. That is not a clean bill of health. That is an untested instrument.

Coding-agent benchmarks get validated at the task level — is this bug realistic, is this repo representative — and almost never at the harness level. Can this rig detect a failure at all? If it cannot, your denominator is fiction, and the percentage above it is decoration.

Here is a canary self-test you can stand up in an afternoon. It uses planted bugs with known ground truth, two prompt phrasings, and four metrics that describe the rig instead of the agent.

Why a high fix rate can survive a broken harness

Two independent things have to hold: the tasks must resemble real work, and the runner must be able to observe failure. Teams audit the first and assume the second.

A concrete case of the second failing: the runner shells out to pytest in a directory where the new test file was never collected, the agent says everything passes, and the scoreboard records a success. Nothing in a standard report distinguishes that from a genuine fix.

The fix is not a better rubric. It is a set of tasks whose correct answer you control before the agent ever touches them.

Step 1: Write a canary manifest

Keep canaries small, boring, and fixed. Each entry names a fixture repo, a planted defect (or null for a clean control), and the single observable signal that proves the defect was detected.

# canary.yaml
version: 1
canaries:
  - id: c-offby-one
    repo: fixtures/range-utils
    planted: 'page_range() drops the final page'
    expected_signal: 'tests/test_range.py::test_last_page'
  - id: c-null-guard
    repo: fixtures/profile-api
    planted: 'None check removed before .strip()'
    expected_signal: 'AttributeError'
  - id: c-clean-a
    repo: fixtures/range-utils
    planted: null
    expected_signal: 'no failure reported'
  - id: c-clean-b
    repo: fixtures/profile-api
    planted: null
    expected_signal: 'no failure reported'

Two clean canaries matter as much as the planted ones. A rig that flags everything is as useless as one that flags nothing, and only the clean controls reveal that.

Step 2: Run canaries through the production adapter

Use the exact adapter your real suite uses. A canary that passes through a special code path proves nothing about production.

The runner below is a sketch, not a finished product — you will replace run_agent with a wrapper around your agent's CLI. Adapt the timeout and the reset step to your fixtures.

# canary_run.py
import json, pathlib, subprocess, time
import yaml

ADAPTER = './run_agent.sh'   # wrapper: <prompt> <repo>

def reset(repo):
    subprocess.run(['git', '-C', repo, 'checkout', '--', '.'], check=True)
    subprocess.run(['git', '-C', repo, 'clean', '-fd'], check=True)

def run_agent(prompt, repo, timeout=900):
    proc = subprocess.run([ADAPTER, prompt, repo],
                          capture_output=True, text=True, timeout=timeout)
    return {'rc': proc.returncode, 'stdout': proc.stdout, 'stderr': proc.stderr}

def main(manifest='canary.yaml', out='canary_runs.json'):
    spec = yaml.safe_load(pathlib.Path(manifest).read_text())
    rows = []
    for c in spec['canaries']:
        repo = c['repo']
        for phrasing in spec.get('phrasings', ['A', 'B']):
            reset(repo)
            prompt = build_prompt(c, phrasing)
            t0 = time.time()
            result = run_agent(prompt, repo)
            rows.append({'id': c['id'], 'planted': c['planted'],
                         'expected': c['expected_signal'], 'phrasing': phrasing,
                         'elapsed_s': round(time.time() - t0, 1), 'result': result})
    pathlib.Path(out).write_text(json.dumps(rows, indent=2))
    return rows

build_prompt is where paraphrase variance lives. Wording A is terse; wording B adds a sentence of context and changes the verb. Both ask for the same observation.

Reset between runs is not optional. If the agent leaves a fixture dirty, your next canary measures the previous run.

Step 3: Report harness metrics, not agent metrics

Now you can count things that mean something.

def summarize(rows):
    planted = [r for r in rows if r['planted']]
    clean   = [r for r in rows if not r['planted']]
    caught  = [r for r in planted
               if signal_in(r['expected'], r['result']['stdout'] + r['result']['stderr'])]
    quiet   = [r for r in clean if r['result']['rc'] == 0]
    cs = len(caught) / len(planted) if planted else float('nan')
    qr = len(quiet) / len(clean) if clean else float('nan')
    return {'canary_sensitivity': cs, 'quiet_rate': qr,
            'n_planted': len(planted), 'n_clean': len(clean)}

Define the four numbers once and keep the definitions fixed across every report you publish:

1. Canary sensitivity (CS) — share of planted defects the rig detected. This is a ceiling on every agent score you will ever publish from it.
2. Quiet rate (QR) — share of clean canaries correctly reported clean. A low QR means your harness leaks noise into results.
3. Paraphrase drift (PD) — canaries whose verdict flips between wording A and B. Drift is a property of the harness, not of the agent.
4. Cost per canary — wall-clock seconds and tokens per canary row. Multiply by your real suite size before you commit to a run.

One caveat: signal_in doing substring matching is a first cut. When it disagrees with what you see by eye, fix the matcher. Do not edit the log.

Step 4: Read the decision table before you quote anything

CS	QR	PD	What the numbers support
1.0	1.0	0.0	Harness passes its own check. Agent scores are interpretable within this rig.
< 1.0	any	any	Every agent score is capped by CS. Fix the rig first; the ranking is unreadable.
any	< 1.0	any	Clean canaries are failing. Agents may be penalized for harness noise.
any	any	> 0.0	Results depend on prompt wording. Freeze the wording and rerun before comparing runs.

The pessimistic row is the useful one. A harness that detects 4 of 6 planted bugs cannot be quoted at a resolution finer than two-thirds, no matter how many decimal places your report prints.

Where a free tier genuinely fits

Running canaries is repetitive and cheap per call, which is exactly the shape of work a free tier absorbs well. MonkeyCode's free model access and free server option are relevant here because the canary pass is throwaway infrastructure: you need a sandbox for the fixture repos and a lot of small model calls that you do not want on your production budget.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

A working order that keeps the spend honest:

1. Run CS, QR, and PD on canaries using free model access. Iterate on the matcher and the prompt wording while the marginal cost is zero.
2. Confirm the rig passes its own check before touching the paid run. A harness that fails its canaries will happily burn budget producing uninterpretable numbers.
3. Move the real suite to paid capacity only after the self-test is green, then rerun a subset of canaries inside the paid environment to check that the environment itself did not change behavior.

Availability details are operator-supplied, not benchmark results. The operator states that the free tier includes access to models at no cost and a free server option, along with a free allowance of up to 10 million tokens. Terms like those change; check the project's current page before you wire it into a pipeline, and do not treat a quoted allowance as a guarantee of duration or throughput.

Limitations and who should not use this

• A canary self-test validates the instrument, not the task set. It says nothing about whether your fixtures resemble production code.
• Four canaries give you a coarse yes/no. Sensitivity estimates get meaningful with dozens, and each addition costs a run.
• Substring signal matching produces false negatives against verbose agents that report a failure without reproducing the exact expected string.
• Do not push proprietary or regulated source code through any third-party free tier unless your own review clears the terms. If you are air-gapped or need audited evaluation, this workflow is not for you; build the harness internally first.
• If your goal is a statistically powered comparison between two agents, this is step zero, not the whole method. Sensitivity and quiet rate come before any interval estimation.

The habit worth keeping

Before any number leaves your terminal, ask what would have to be true for the harness to report a false success. Then plant that condition and watch whether the rig notices.

If you run this on your own suite, the interesting failure is paraphrase drift — two wordings, two verdicts. That is the one that quietly invalidates a quarter of comparison posts. If you find a pair that flips, share the wording difference in the comments; a reproducible harness bug is worth more than another leaderboard.