Red Before Green: A Probe Gate for Free-Tier Model Output

Tuesday pairing hour, forty minutes left on the clock. The model proposed a four-line fix for a rounding bug in a price calculation, and it also wrote the test that covered the fix.

The diff looked reasonable. The test went green on the first run. That is usually the moment a session ends early, with a patch that nobody has actually challenged.

The senior on the call did not read the diff. The senior asked one question: does that test fail without your fix?

Nobody knew. The test had been written after the patch, against the patched tree. It had never been red.

Three dead ends that ate the hour

Before the useful question landed, the session tried three things that felt like progress and produced no decision.

1. Reading the diff for correctness. A four-line change is short enough to read and long enough to misread. Reviewing it produced opinions, not evidence, and opinions do not survive a code review two days later.
2. Re-prompting with more context. A second attempt with the surrounding module pasted in produced a larger diff and the same unanswered question. More context did not make the probe load-bearing.
3. Re-running the suite and watching it pass. Green on the patched tree is exactly what a test that asserts nothing will do. Passing is the weakest possible signal here.

None of these answered the only question that mattered: if the patch were reverted, would anything in the repository notice?

The question that ends an argument in one run

Does the probe fail on the tree without the patch? That is decidable. It takes one checkout, one command, and no judgement about code style or author intent.

A probe that has never been red is a description, not a test

This is the rule the session kept. Not "write tests first" as a ritual, but a narrower and checkable property: a probe must be red on the unfixed tree and green on the fixed tree, with the same inputs on both sides.

If the probe is green on both trees, it is documenting existing behaviour. If it is red on both trees, the patch is incomplete or the probe is wrong. Only one of those four combinations is a result.

The artifact: a probe gate in about sixty lines

The gate below runs a probe against two checkouts, hashes a frozen input set in each, repeats every run to catch flake, and writes a JSON report. It has no dependencies outside the standard library.

#!/usr/bin/env python3
"""probe_gate.py - a probe must be red before a patch and green after it.

Exit codes: 0 gate passed, 1 gate failed, 2 usage/environment error.
"""
from __future__ import annotations

import argparse
import hashlib
import json
import shlex
import subprocess
import sys
from pathlib import Path


def digest(paths: list[Path]) -> str:
    h = hashlib.sha256()
    for p in sorted(paths):
        h.update(p.as_posix().encode())
        h.update(p.read_bytes())
    return h.hexdigest()[:12]


def run(cmd: list[str], cwd: Path) -> tuple[int, str]:
    proc = subprocess.run(cmd, cwd=cwd, capture_output=True, text=True, timeout=900)
    return proc.returncode, (proc.stdout + proc.stderr)[-2000:]


def main() -> int:
    ap = argparse.ArgumentParser()
    ap.add_argument("--before", required=True, type=Path, help="checkout without the patch")
    ap.add_argument("--after", required=True, type=Path, help="checkout with the patch")
    ap.add_argument("--probe", required=True, help="e.g. 'pytest -q tests/test_rounding.py'")
    ap.add_argument("--frozen", nargs="*", default=[],
                    help="files whose bytes must match in both trees")
    ap.add_argument("--repeat", type=int, default=2, help="runs per tree, to catch flake")
    ap.add_argument("--report", type=Path, default=Path("probe-report.json"))
    args = ap.parse_args()

    cmd = shlex.split(args.probe)
    frozen = [Path(f) for f in args.frozen]

    before_digest = digest([args.before / f for f in frozen]) if frozen else "-"
    after_digest = digest([args.after / f for f in frozen]) if frozen else "-"

    report: dict[str, object] = {
        "probe": args.probe,
        "frozen_digest_before": before_digest,
        "frozen_digest_after": after_digest,
        "runs": [],
    }

    verdict = "pass"
    if before_digest != after_digest:
        verdict = "fail: frozen inputs differ between trees"

    before_codes, after_codes = [], []
    for _ in range(args.repeat):
        code, tail = run(cmd, args.before)
        before_codes.append(code)
        report["runs"].append({"tree": "before", "exit": code, "tail": tail})
    for _ in range(args.repeat):
        code, tail = run(cmd, args.after)
        after_codes.append(code)
        report["runs"].append({"tree": "after", "exit": code, "tail": tail})

    if len(set(before_codes)) != 1 or len(set(after_codes)) != 1:
        verdict = "fail: flaky probe (exit codes differ across repeats)"
    elif before_codes[0] == 0:
        verdict = "fail: probe already passes without the patch"
    elif after_codes[0] != 0:
        verdict = "fail: probe still red after the patch"

    report["verdict"] = verdict
    args.report.write_text(json.dumps(report, indent=2))
    print(verdict)
    print(f"report: {args.report}")
    return 0 if verdict == "pass" else 1


if __name__ == "__main__":
    sys.exit(main())

Running it

The two trees are cheap. A git worktree gives you the unpatched checkout without stashing anything.

git worktree add ../tree-before HEAD

python probe_gate.py \
  --before ../tree-before \
  --after . \
  --frozen fixtures/cart.json src/pricing.py \
  --probe "python -m pytest tests/test_rounding.py -q" \
  --repeat 2 \
  --report probe-report.json

--frozen is the part people skip. If the fixture the probe reads is different in the two trees, the gate fails immediately, because a probe that runs against two different worlds is not comparing a patch to its absence.

A log shape worth copying

The table below shows the columns the gate produces. The values are illustrative, not measurements from a specific deployment.

Probe	Tree	Exit	Reading
rounding	before	1	red, as required
rounding	before	1	stable across repeat
rounding	after	0	green
rounding	after	0	stable across repeat
tax-total	before	0	rejected: never red

Four of five rows are one probe doing its job. The fifth row is the common failure: a test that the model wrote to describe existing behaviour, presented in the same diff as an assertion about the fix.

Routing questions in a pairing hour

Not every question in a session deserves the same responder. The routing table below is the practical version of what the senior did on the call.

Question	Cheapest honest source	Reason
What does this code do now?	Run it	Execution beats explanation
What inputs might break it?	Model, freely	Hypothesis generation is cheap and wrong answers cost little
Is this probe load-bearing?	The unpatched tree	One run, binary answer
Is this acceptable to ship?	A human owner	No model output carries accountability
Is the failure environmental?	The frozen digest + a clean container	Distinguishes code from machine

The second row is where delegate-heavy workflows get their value. The third row is where they get their safety.

Where free model access and a free server change the economics

Probing is repetitive, and repetition is exactly what a free tier is good for. In this workflow the model's job is narrow: propose candidate probes for a failure that a human has already described. Each candidate is then run twice, on two trees, by a machine nobody has to pay for. MonkeyCode's operator states that the project offers free model access and a free server option, which is what makes that proposal-then-discard loop affordable rather than an expensive per-call ritual. I have not verified current quotas, limits, retention, or hardware, and a large free token allowance circulating in promotion is a marketing number until the project's own current terms confirm it. Check the docs before you plan a month around it.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

One caution that belongs next to the disclosure: a free server is a place to run experiments, not a place to keep the only copy of an artifact. Treat the container as disposable and push the JSON report back to your own repository or CI storage.

Limitations, and who should skip this

• The gate cannot tell you whether the probe tests the right thing. A red-before, green-after probe that asserts a log line is technically valid and practically useless. Only review catches that.
• Frozen inputs are a human decision. The gate verifies that the listed files match; it cannot know which files are causally relevant.
• Flake detection costs time. --repeat 2 doubles every run. On slow suites, drop to one repeat and accept the risk.
• It does not fit nondeterministic failures. Race conditions, timing bugs, and visual regressions need different instruments.
• Skip it entirely if you cannot define acceptance criteria. A team that cannot write down what "fixed" means will only produce rows in a report.
• Skip it for dependency bumps and generated code. There is no meaningful unpatched tree to compare against.

What the hour kept

The session kept a rule, not a tool: no model-authored test enters the review until the gate has shown it red on the tree without the patch. It costs about two minutes per probe and it removes the single most common way a pairing session produces confident, empty evidence.

The senior's last comment before the call ended was that the gate is boring, which is the point. Boring verification is what makes fast, generous delegation safe. If you want to try the red-before-green loop on a machine you do not have to provision yourself, the free server option is where this workflow was designed to run.