Green Is Not Enough: A 60-Minute AI Fix Verification Workshop

At 2 a.m. a test turned red in CI. A developer pasted the failure into a chat window, received a five-line patch, and merged it once the same test went green. Forty minutes later the next pipeline job failed, because the patch had quietly deleted the assertion that actually mattered. The test was green. The system was not.

This article is a one-hour workshop outline for a different habit: treat every AI-generated fix as a hypothesis, then run it through a three-pass ritual before it reaches a shared branch. Participants leave with a script they can rerun, a trap they can teach, and a loop that runs on free compute. The workshop uses free model access from MonkeyCode, an open-source project, and its free server option for the final exercise. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The three-pass ritual

The core idea is borrowed from test-driven development. A fix is not a patch; it is a claim that a failing test becomes green without breaking anything else. The claim deserves three passes.

Pass one makes the failure reproducible. Participants run the failing suite by hand and write down what the failure actually says; copying a traceback is not the same as reading it. Pass two asks a model for a minimal diff, with one hard constraint: the diff must not touch the test file. Pass three applies the diff in a throwaway copy, reruns the suite, and then a human reads the diff aloud. The last step is the one most teams skip, and it is the one that catches the 2 a.m. story.

The artifact

The workshop centers on a small script, around sixty lines, that encodes the ritual. One config, one prompt, one verdict. It is not a review tool; it is a decision gate that prints a JSON line so a team can collect evidence.

# verify_fix.py — a three-pass gate for AI-generated fixes
import json
import os
import subprocess
import sys
import time
import urllib.request


def call_model(prompt: str) -> str:
    base_url = os.environ["MODEL_URL"]
    api_key = os.environ["MODEL_KEY"]
    model = os.environ["MODEL_NAME"]
    payload = {
        "model": model,
        "messages": [{"role": "user", "content": prompt}],
        "temperature": 0.2,
    }
    request = urllib.request.Request(
        base_url.rstrip("/") + "/chat/completions",
        data=json.dumps(payload).encode(),
        headers={
            "Content-Type": "application/json",
            "Authorization": "Bearer " + api_key,
        },
    )
    with urllib.request.urlopen(request, timeout=60) as response:
        body = json.load(response)
    return body["choices"][0]["message"]["content"]


def run(command: str) -> tuple[int, str]:
    process = subprocess.run(command, shell=True, capture_output=True, text=True)
    return process.returncode, process.stdout + process.stderr


def main() -> None:
    workdir = sys.argv[1]
    test_command = sys.argv[2]

    red, _ = run(f"cd {workdir} && {test_command}")
    if red == 0:
        print(json.dumps({"phase": "skipped", "reason": "already green"}))
        return

    with open("failure.txt", encoding="utf-8") as handle:
        failure = handle.read()
    with open("task.txt", encoding="utf-8") as handle:
        task = handle.read()

    prompt = (
        "A test fails in a Python repository.\n"
        f"FAILURE OUTPUT:\n{failure}\n"
        f"TASK DESCRIPTION:\n{task}\n"
        "Return only a unified diff that fixes the code.\n"
        "Do not modify the test file and do not delete assertions."
    )
    started = time.time()
    patch = call_model(prompt)

    if "test_" in patch:
        print(json.dumps({"phase": "rejected", "reason": "patch touches the test file"}))
        return

    with open("candidate.patch", "w", encoding="utf-8") as handle:
        handle.write(patch)

    ok, _ = run(f"cd {workdir} && git apply --check candidate.patch")
    if ok != 0:
        print(json.dumps({"phase": "unparsable", "elapsed_s": round(time.time() - started, 2)}))
        return

    run(f"cd {workdir} && git apply candidate.patch")
    green, output = run(f"cd {workdir} && {test_command}")
    print(json.dumps({
        "phase": "verified" if green == 0 else "rejected",
        "elapsed_s": round(time.time() - started, 2),
        "tail": output[-400:],
    }))


if __name__ == "__main__":
    main()

The script reads three environment variables: MODEL_URL and MODEL_KEY, for any OpenAI-compatible endpoint, and MODEL_NAME, for the model identifier. It expects failure.txt with captured test output and task.txt with a one-sentence description of the intended behavior. It runs a command in a clean checkout; if the suite is already green it prints skipped and stops. A red suite produces a prompt with a strict instruction to return only a diff. The script refuses patches that touch the test file, rejects patches that do not apply cleanly, and records the elapsed time and the tail of the final run.

The worked example

The example is deliberately trivial, because the point is the ceremony, not the difficulty. A file called price.py contains a single function.

def total_with_tax(price: float, tax: float) -> float:
    return price * (1 + tax)

A test file asserts the total with tax rounds to cents.

def test_total_with_tax_rounds_to_cents():
    assert total_with_tax(19.99, 0.0875) == 21.74

The test fails, because binary floating-point numbers rarely produce exact cents. The facilitator captures the failure, writes task.txt, and lets the group run the script. The honest fix wraps the result in round.

def total_with_tax(price: float, tax: float) -> float:
    return round(price * (1 + tax), 2)

The script prints verified. Then comes the trap. A second candidate patch also passes the single test, but it special-cases the exact price and rounds everything else.

def total_with_tax(price: float, tax: float) -> float:
    if price == 19.99:
        return 21.74
    return round(price * (1 + tax), 2)

The script prints verified again, because the test only covers one input. The lesson lands in the gap between those two runs: an exit code of zero proves consistency with one test, not correctness. The facilitator asks the group to read each patch aloud, and the second patch fails the audit instantly, because a hard-coded value is not a rule.

The sixty minutes

The first ten minutes belong to reproduction. Participants clone a tiny repository, run the failing test, and write failure.txt in their own words. The next fifteen belong to the script; the group runs verify_fix.py against the example and records the JSON verdict. The next twenty belong to the trap round, with three candidate patches, each with a different flaw. The diff audit separates them; the script alone cannot. The final fifteen belong to persistence: the same loop moves to the free server option, so a scheduled run can watch a branch overnight instead of depending on a laptop. Hosting details drift quickly, so the facilitator prints them live rather than hard-coding them in the handout.

Limits and the wrong audience

A single green run is a sample size of one. The script does not catch a fix that passes a weak test and breaks a strong one, and it cannot judge style, security, or intent. Free-tier capacity has ceilings; the script logs tokens and elapsed time precisely, so a team can notice when a task is too large for the free option. Security-sensitive patches need a sandbox and a human with authority, not a chat-completion reply.

Teams without a test suite should skip this workshop, because the ritual has nothing to hold on to. Teams whose CI is already red for unrelated reasons will only certify noise. And individuals who already read every diff aloud will find the drill redundant, which is a good problem to have.

The artifact runs against any OpenAI-compatible endpoint, including the same free model access from MonkeyCode mentioned earlier; the only setup is the three environment variables. The natural next artifact is a bot that runs the same three passes on every pull request. That workshop starts the same way, with a test that fails first.