Diff the AI's Test Draft Before You Adopt It

An AI-drafted characterization test is a hypothesis. Adoption without verification is a gamble. Draft, run, diff, adopt. That four-step loop is the whole job. Here is a working harness for it.

Why a Draft Looks Correct

A messy module has no tests. You want to refactor it safely. You ask a free model to draft characterization tests. The output looks professional. It reads like truth. Truth requires evidence.

Models interpolate patterns. They do not execute your code. Their training data contains thousands of tax functions. None of them is yours.

Draft tests fail in three predictable ways:

1. Invented expectations. The model asserts values no real call ever produced.
2. Bug normalization. The model "fixes" what it sees. The draft encodes intent, not behavior.
3. Context blindness. The draft uses inputs your code never receives.

All three produce a green test. Green is not evidence. Green only means the draft agrees with itself. Every failure mode survives pytest. The suite passes. The suite is wrong. The diff loop exists because passing is cheap.

Step 1: Record Fixtures from Current Behavior

Fixtures are recorded observations, not opinions. Sample real inputs. Capture real outputs. Store them as JSON Lines.

Choose inputs that touch each branch. Coupon present. Coupon absent. Empty cart. One item. Non-integer quantity, if the schema allows it. Five varied samples catch more deltas than fifty similar ones.

# record_fixtures.py
import json
import order

samples = [
    {"items": [{"price": 10.0, "qty": 2}], "coupon": None},
    {"items": [{"price": 4.99, "qty": 1}], "coupon": "SAVE10"},
    {"items": [], "coupon": None},
]

with open("fixtures.jsonl", "w") as fh:
    for sample in samples:
        row = {"input": sample, "output": order.total_with_tax(sample)}
        fh.write(json.dumps(row) + "\n")

Step 2: Draft Tests with a Free Model

Drafting is an iteration problem. Most candidates get discarded. Cheap iteration matters here. This drafting loop runs on MonkeyCode's free model access through the free server option. No local GPU needed. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The draft prompt needs a hard constraint. Verifiable tests require a predictable shape.

Write each test case as:
assert total_with_tax(<input literal>) == <expected literal>
No fixtures. No mocks. No loops.

That shape is what makes automated review possible. Free the format, and verification breaks.

Generate three draft candidates. Never settle for the first. Each candidate costs almost nothing. Run all three through the harness. Merge the passing cases into one file. That file is the candidate suite.

Step 3: Diff Draft Expectations Against Reality

The harness parses the draft's assertions. It compares each expected value to a recorded fixture. It prints one verdict per case.

# verify_draft.py
import ast
import json
import subprocess
import sys
from pathlib import Path


def draft_expectations(path):
    tree = ast.parse(Path(path).read_text())
    rows = []
    for node in ast.walk(tree):
        if not isinstance(node, ast.Assert):
            continue
        cmp = node.test
        if not isinstance(cmp, ast.Compare):
            continue
        left, right = cmp.left, cmp.comparators[0]
        if isinstance(left, ast.Call) and isinstance(right, ast.Constant):
            rows.append((left.args[0], right.value))
    return rows


def load_fixtures(path="fixtures.jsonl"):
    return [json.loads(line) for line in Path(path).read_text().splitlines()]


def main():
    observed = {
        json.dumps(f["input"], sort_keys=True): f["output"]
        for f in load_fixtures()
    }
    print(f"{'input':<46} {'draft':<14} {'observed':<14} verdict")
    for inp, expected in draft_expectations("test_order_draft.py"):
        key = json.dumps(inp, sort_keys=True)
        actual = observed.get(key)
        if actual == expected:
            verdict = "MATCH"
        elif key in observed:
            verdict = "DELTA"
        else:
            verdict = "NO_FIXTURE"
        print(f"{json.dumps(inp)[:46]:<46} {expected!s:<14} {str(actual):<14} {verdict}")

    run = subprocess.run(
        [sys.executable, "-m", "pytest", "test_order_draft.py", "-q"],
        capture_output=True,
        text=True,
    )
    print(f"pytest exit code: {run.returncode}")


if __name__ == "__main__":
    main()

Here is the output shape on a messy module:

input                                            draft      observed   verdict
{"items": [{"price": 10.0, "qty": 2}], "coupon": null}  21.6       21.6       MATCH
{"items": [{"price": 4.99, "qty": 1}], "coupon": "SAVE10"}  5.39  4.85       DELTA
{"items": [], "coupon": null}                             0.0        0.0       MATCH
pytest exit code: 1

The DELTA row is the model guessing intent instead of reading behavior. The model inferred that a coupon discounts the total. The real code discounts the subtotal first. Both are plausible. Only one is true.

Inspect every DELTA manually. Read the function. Trace the inputs. Ask one question: did the model misread the code, or does the code betray its own intent? The answer decides the fix. If the model is wrong, delete the case. If the code is wrong, write a spec test and annotate the bug.

The parser only reads literal comparisons. Other shapes are skipped silently. Keep the prompt constraint strict, or extend the parser.

Step 4: Apply the Mutation Gate

A passing draft test can assert nothing useful. Prove usefulness the cheap way. Break behavior on purpose. The suite must fail.

# Change the tax rate. Nothing else.
sed -i 's/0.08/0.09/' order.py
python -m pytest test_order_draft.py -q   # must show failures
git checkout order.py

One intentional break. Multiple failures. That is the mutation gate. Run one break per gate pass. Two breaks can hide each other.

Reading the Verdicts

Verdict	Meaning	Action
MATCH	Draft equals a real observed output.	Keep the test.
DELTA	Draft disagrees with reality.	Inspect. The draft may be wrong. The code may be buggy.
NO_FIXTURE	No recorded input reached this path.	Record a fixture or drop the test.

Expect DELTA verdicts in a messy module. They are not noise. They are the review. A draft with zero deltas is either excellent or untested. Assume untested. A DELTA usually fails pytest. That failure is the review working.

Adoption rules before you merge:

• Every adopted test has a MATCH verdict.
• Every MATCH has a fixture.
• The mutation gate failed once per intentional break.
• No test entered the suite without a verdict.

Mechanical rules beat taste. Encode them in CI if you can. The harness is a script. Scripts do not get tired.

Then Make the Smallest Safe Change

Verified tests are your behavior contract. Make one move. Commit the suite first.

# one move: extract the discount decision
def discount_rate(order):
    return 0.9 if order.get("coupon") == "SAVE10" else 1.0


def total_with_tax(order):
    subtotal = sum(i["price"] * i["qty"] for i in order["items"])
    return round(subtotal * 1.08 * discount_rate(order), 2)

Run the suite. Zero failures means behavior held. Commit the suite. Commit the move. Do not merge the two commits.

Limitations and Who Should Skip This

This loop verifies current behavior. It never verifies correct behavior. If the code is the bug, characterization tests protect the bug. Write spec tests instead.

Skip this workflow when:

• The code is greenfield. Write the spec first.
• The refactor must change behavior. Characterization locks the old contract.
• The draft will be merged unreviewed. A draft is a hypothesis.

Fixtures decay. Inputs change. Recapture them when the suite goes green too easily. A stale fixture is a false memory.

The free-server side is an availability claim, not a guarantee. Model identity, quotas, and uptime change. None are verified here. The harness works with any drafting source. Verification stays local and controllable. That split is the point.

The Score That Matters

Run the loop on your messiest module. Count the DELTA verdicts. The count is your review quality score. Post it. Compare notes with other reviewers.

Adoption sequence:

1. Record fixtures.
2. Draft three candidates.
3. Diff every expectation.
4. Kill the DELTA rows.
5. Run the mutation gate.
6. Commit the suite.
7. Make one move.
8. Run again. Commit.