The Pairing Wrote a Failing Probe Before Any Free Model Could Patch

The pairing kept one rule when a green suite hid a defect: a senior-authored probe had to fail first. Coverage percentages did not count as evidence, because they never named the invariant the path violated. The session treated existing tests as a map of prior beliefs, not as an oracle for this bug. Only after that failing probe existed did the pair allow a free model run on an allowlisted server.

Green coverage is a memory, not a verdict

A modern agent can raise line coverage while leaving the real contract untouched, especially on branches that serialize money, identity, or retries. The suite still reports success because earlier engineers asserted shapes, status codes, and happy paths, not the missing invariant. Pairing with a senior is useful here because the senior can name what the suite never said. The junior and the model cannot recover that missing name from passing tests or from a coverage badge alone.

Recent public debate about vibe coding versus engineering often collapses into taste, slogans, and tool loyalty. This pairing ignored that noise and instead asked a narrower question about what counted as evidence. A patch was not engineering work until a human could point at one failing check the suite had omitted. Without that check, a free model would optimize for the tests that already existed and would look productive while remaining wrong.

Questions the senior asked before any model ran

The senior did not start by opening an editor or by pasting a stack trace into a chat panel. The senior walked the failing production path on a whiteboard and forced every claim into a named object. The pairing recorded four questions in the session notes before any assistant call was made. Those answers later became fields on a small probe card checked into the branch.

1. What sentence states the invariant in domain language, without mentioning mocks, fixtures, or framework glue?
2. What input is the smallest counterexample that currently returns a wrong value or performs a forbidden side effect?
3. Which assertion in the existing suite would stay green even when that counterexample is replayed against the handler?
4. What side effect must the new probe forbid, such as a second charge or a dropped idempotency key?

Each question required an answer that named a path, a symbol, or a concrete payload. Vague answers such as the payment flow or the tests around checkout were rejected and rewritten on the spot. The pairing did not proceed while any of the four answers still pointed at a directory instead of a file or a value.

Dead ends the session closed

The first dead end was asking a model to write the new tests before a human had named the invariant. That order looks efficient, yet it lets the model invent assertions it can already satisfy. The second dead end was treating a high coverage badge as permission to ship, even though the counterexample never entered the suite. The third dead end was booting a shared server early so the agent could explore, which mixed probe evaluation with unbounded shell access.

Those three paths were written into the session notes as closed, not as later options. Closing them mattered because each path would have produced a plausible transcript and a green job. The pairing needed a result that could still fail after the existing suite had passed. The senior kept the conversation on that single gap until the probe card had a failing command.

The decision the pair kept

The kept decision was a probe gate, not a broader process rewrite and not a new chat persona. A human had to commit one failing contract probe that encoded the missing invariant in ordinary test code. A model could propose a patch only against an explicit file list, and only after that probe failed on the current branch. A free server could run only allowlisted commands that executed the probe, not an open shell.

The pair then used MonkeyCode only as the later execution lane, after the failing probe and the command allowlist already existed. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source coding assistant with operator-supplied free model access and a free server option, which the pairing treated as a constrained runner rather than as an author of tests. Free model access was reserved for proposing a patch against the frozen file list, not for inventing coverage. The free server option was reserved for running allowlisted probe commands, not for open-ended exploration of the repository.

A probe card the branch can carry

The pairing stored the answers as a short Markdown card beside the probe, so later reviewers could see the invariant without replaying the conversation. The card below is a proposed template, not a claim about any product file format. Teams can keep it in pairing/probe-card.md or fold the same fields into the test docstring.

# Probe card: double charge on retry

- Invariant: a paid idempotency key must not create a second capture.
- Counterexample: POST /charges with key K1, then replay the same body after a 504.
- Existing suite gap: tests/test_charges_api.py asserts HTTP 201 and JSON shape only.
- Forbidden side effect: ledger.capture_count(K1) must remain 1.
- Probe command: pytest -q tests/test_contract_probe.py::test_replay_does_not_double_capture
- Writable files: app/charges.py, app/ledger.py
- Frozen files: tests/test_charges_api.py, pairing/probe-card.md
- Allowlisted commands: python -m pytest, python -m compileall app

The card is small on purpose, because a long narrative invites the next session to skip reading it. Reviewers should be able to reject a patch that touches a frozen file without debating model quality. The probe command should be one line that a gate script can run twice: once before the patch, and once after. If the before run does not fail, the gate has nothing honest to measure.

Contract probe the senior wrote by hand

The example below is labeled as an unexecuted illustration of a missing invariant around charge replay. It does not pretend to be production billing code, and it does not call any hosted model. The important property is that the existing suite can stay green while this probe fails on the same handler.

# tests/test_contract_probe.py
# Proposed local example. Not executed against a live payment network.

from app.charges import ChargeHandler
from app.ledger import InMemoryLedger


def test_replay_does_not_double_capture():
    ledger = InMemoryLedger()
    handler = ChargeHandler(ledger)
    body = {"idempotency_key": "K1", "amount_cents": 1500, "currency": "USD"}

    first = handler.create_charge(body, upstream_status=201)
    replay = handler.create_charge(body, upstream_status=504)

    assert first.status in {"captured", "pending"}
    assert replay.status in {"captured", "pending", "replayed"}
    assert ledger.capture_count("K1") == 1
    assert ledger.captures_for("K1")[0].amount_cents == 1500

A matching handler sketch helps the pairing show why coverage can lie. The buggy version increments the ledger on every retry path that looks new to the HTTP layer. The existing API test never replays a 504, so it never sees the second capture. The probe makes that gap visible without asking a model to notice it.

# app/charges.py  (buggy sketch for the pairing, not a library)

class ChargeHandler:
    def __init__(self, ledger):
        self.ledger = ledger

    def create_charge(self, body, upstream_status=201):
        key = body["idempotency_key"]
        # Bug: retry after 504 is treated as a fresh capture.
        if upstream_status == 504 or not self.ledger.has(key):
            self.ledger.capture(key, body["amount_cents"])
            return type("Result", (), {"status": "captured"})()
        return type("Result", (), {"status": "replayed"})()

Numbered pairing workflow

The workflow is a gate, so each step has an exit that does not spend model time. The commands are local proposals that any repository can adopt. Replace tool names with the team runner if the pair already has one.

1. Write the probe card and the failing test in the same change, with frozen files listed explicitly.
2. Run the probe once on the current branch and store the failing output beside the card.
3. Refuse to continue if the probe is green, because then the invariant is not actually missing.
4. Open a constrained assistant session only after the failing output exists in git or in the session notes.
5. Allow the model to edit only the writable files, then rerun the same probe command with no extra flags.
6. Keep the patch only when the probe turns green and the frozen files still match git diff --stat.

# Proposed local gate. Label: unexecuted example.
set -euo pipefail
PROBE="tests/test_contract_probe.py::test_replay_does_not_double_capture"
CARD="pairing/probe-card.md"
ALLOW="pairing/allowlist.txt"

test -f "$CARD"
git diff --name-only -- "$CARD" | grep -q . && echo "card should be committed before model work" && exit 1

if pytest -q "$PROBE"; then
  echo "probe is green; the gate has no defect to measure"
  exit 1
fi

echo "probe failed as required; model work may start on writable files only"

After a patch is proposed, the pair reruns one allowlisted command and inspects the file list. The review is boring on purpose. Boring reviews are cheaper than arguing about whether the model understood checkout.

# After the proposed patch
pytest -q tests/test_contract_probe.py::test_replay_does_not_double_capture
git diff --name-only > /tmp/touched.txt
# fail if any frozen file appears in /tmp/touched.txt

Allowlist for the free server lane

The pairing did not treat a free server as a general workstation. It treated the server as a place to evaluate one command family. The allowlist below is a proposed text file, not a documented product interface. A wrapper can reject any argv that is not a prefix match.

# pairing/allowlist.txt
python -m pytest tests/test_contract_probe.py
python -m pytest tests/test_contract_probe.py::test_replay_does_not_double_capture
python -m compileall app
git diff --name-only
git diff --stat

# scripts/allowlist_exec.py
# Proposed local wrapper. Unexecuted example.

from pathlib import Path
import subprocess
import sys

allowed = [
    line.strip()
    for line in Path("pairing/allowlist.txt").read_text().splitlines()
    if line.strip() and not line.startswith("#")
]
request = " ".join(sys.argv[1:])
if request not in allowed:
    raise SystemExit(f"command not allowlisted: {request}")
raise SystemExit(subprocess.call(sys.argv[1:]))

The wrapper is intentionally strict, including exact command strings instead of regex. Strictness prevents a model from appending -k filters, extra paths, or shell metacharacters that the senior never approved. If the pair needs a new command, a human adds it to the allowlist in a separate change. That extra commit is part of the evidence, not ceremony.

Decision table for the gate

Observation on the branch	Action the pairing takes	Why the action is kept
Existing suite green, probe missing	Stop and write the probe card	No invariant has been named
Probe green on current code	Rewrite the counterexample	The gate cannot measure a defect
Probe red, frozen file dirty	Revert the frozen file	The suite gap must remain visible
Probe red, writable files only	Allow a constrained model pass	The patch has a failing check to beat
Probe green after patch, allowlist intact	Keep the diff for human review	The missing invariant is now asserted
Command not in allowlist	Reject the server run	Exploration is not evaluation

The table is the artifact reviewers can apply without joining the original pairing session. It also prevents a later session from reopening the three dead ends under a friendlier name. If a row does not match, the pair writes a new card instead of stretching this one. Stretching cards is how green suites become fiction again.

Limitations

The probe is only as honest as the invariant the senior named, and seniors are sometimes wrong about production. A precise probe can still miss a neighboring defect, such as a currency mismatch that the replay never included. An allowlist can be copied too widely, after which the free server lane becomes an ordinary shell with extra steps. Free model access and a free server option are availability claims for this workflow, not a promise of capacity, latency, model identity, or lasting uptime.

The gate also does not replace code review, threat modeling, or a staging replay against real vendor sandboxes. It only blocks a common failure mode in which an assistant looks busy while the original suite stays silent. Teams that need quantitative comparisons should measure their own probe fail-then-pass rate on their own branches. This article does not offer unpublished benchmarks, model names, token quotas, or hardware details.

Who should not use this approach

Solo work without anyone who can name a domain invariant should not pretend this gate exists. Incident commanders under time pressure should not pause a live outage to invent a probe card format. Teams whose policy forbids remote execution should not route secrets, customer payloads, or production credentials through a free server lane. Security-sensitive patching that can change authorization should stay on an internal runner with a reviewed identity, even if a probe already fails locally.

The pairing format also wastes time on purely cosmetic diffs, dependency bumps, and generated code where no business invariant is at stake. In those cases a linter or a compiler is already a better oracle than a senior interview. Use the probe gate when a green suite can hide a second charge, a dropped key, or another contract the tests never wrote down. Leave it idle when the change cannot produce that class of lie.

Readers who already keep a failing probe in git can try the same order with free model access and a free server option, then keep or discard the patch based on that probe alone.