A Contract Harness for AI-Generated Patches: Review Behavior, Not Vibes

Why this is worth reading

An AI-generated patch can pass your existing unit tests and still be wrong in a way the suite never expressed. When you ask a free model to change a small CLI surface—a duration formatter, a flag parser, or a config key migration—the model often guesses the surrounding behavior. It may produce the happy path output but return the wrong exit code, print the error to stdout instead of stderr, or reject an argument that previously worked. Unit tests catch many of those mistakes if you have already encoded them. They catch almost none of them when the behavior lives only in an informal command, an onboarding document, or the memory of the maintainer who last touched the file.

This guide gives you a cheap way to make that implicit knowledge explicit. You will build a small Python contract harness that applies an AI-generated patch to a temporary git worktree, runs the patched command against concrete input/output examples, and returns a pass or hold decision before any human spends review time. The artifact is deliberately tiny: one script, one JSON contract file, and three shell commands. It is not a replacement for your test suite, but it is a fast gate for exactly the class of free-model mistakes that look believable in a diff.

The cost constraint

A contract harness is useful only if you are willing to run it often, including against several candidate revisions in a row. That constraint pushes you toward a zero-cost loop. MonkeyCode offers free model access and a free server option, so you can generate candidate revisions and host the review harness without spending paid CI minutes. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The workflow below is product-neutral: it only needs a git repository, Python 3, and an executable target.

The free tier is appropriate for controlled experiments. Do not put secrets, customer data, or production-only environment values into the contract cases, and treat the server endpoint as untrusted the same way you would treat any shared runner. If you later move the harness to a private CI worker, the same script and contract file should follow without changes.

What the harness checks

A contract case is a behavioral statement, not a unit test. For each case, you record the command arguments, optional stdin, expected exit code, required stdout fragments, forbidden stderr, and a timeout. The harness runs the same command against each case after applying the candidate patch. A case passes only if every recorded fact matches.

This is intentionally narrower than a full test suite. It does not care how the function is implemented, how many lines the patch touches, or whether the diff looks idiomatic. It cares about externally observable behavior: the process exit status, the text it emits, and the time it takes to answer.

The script

Save the following as contract_harness.py:

import argparse
import json
import subprocess
import sys
import tempfile
import time
from pathlib import Path

def load_contract(path):
    with open(path, 'r', encoding='utf-8') as f:
        spec = json.load(f)
    if 'command' not in spec or 'cases' not in spec:
        raise ValueError('contract must define command and cases')
    return spec

def run_one(command, case):
    started = time.monotonic()
    timeout = case.get('timeout_seconds', 5)
    try:
        proc = subprocess.run(
            command + case.get('args', []),
            input=case.get('stdin', ''),
            text=True,
            capture_output=True,
            timeout=timeout,
        )
    except subprocess.TimeoutExpired:
        return {
            'id': case['id'],
            'status': 'FAIL',
            'reason': 'timed out after {}s'.format(timeout),
            'duration': round(time.monotonic() - started, 2),
        }
    checks = []
    expected_code = case.get('exit_code', 0)
    if proc.returncode != expected_code:
        checks.append('exit code {} != {}'.format(proc.returncode, expected_code))
    for needle in case.get('stdout_contains', []):
        if needle not in proc.stdout:
            checks.append('missing stdout fragment: {!r}'.format(needle))
    for needle in case.get('stderr_contains', []):
        if needle not in proc.stderr:
            checks.append('missing stderr fragment: {!r}'.format(needle))
    if case.get('stderr_empty', False) and proc.stderr.strip():
        checks.append('unexpected stderr: {!r}'.format(proc.stderr[:120]))
    passed = len(checks) == 0
    return {
        'id': case['id'],
        'status': 'PASS' if passed else 'FAIL',
        'reason': '; '.join(checks),
        'duration': round(time.monotonic() - started, 2),
    }

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('--repo', required=True)
    parser.add_argument('--patch', default=None)
    parser.add_argument('--contract', required=True)
    args = parser.parse_args()

    repo = Path(args.repo).resolve()
    spec = load_contract(args.contract)
    command = spec['command'].split()

    with tempfile.TemporaryDirectory() as tmp:
        worktree = Path(tmp) / 'worktree'
        subprocess.run(
            ['git', '-C', str(repo), 'worktree', 'add', '--quiet', '--detach', str(worktree), 'HEAD'],
            check=True,
        )
        if args.patch:
            patch = Path(args.patch).resolve()
            try:
                subprocess.run(['git', 'apply', str(patch)], cwd=worktree, check=True)
            except subprocess.CalledProcessError as exc:
                print('Patch does not apply cleanly: {}'.format(exc))
                sys.exit(2)
        results = [run_one(command, case) for case in spec['cases']]
        for result in results:
            print(result['status'], result['id'], result['duration'])
            if result['reason']:
                print('  reason:', result['reason'])
        passed = sum(1 for result in results if result['status'] == 'PASS')
        total = len(results)
        print('{}/{} contract cases passed'.format(passed, total))
        if passed < total:
            sys.exit(1)

if __name__ == '__main__':
    main()

Define a behavior contract

Create contracts/format_duration.json. In this example, the command is python scripts/format_duration.py, and the contract pins three observable facts:

{
  "command": "python scripts/format_duration.py",
  "cases": [
    {
      "id": "zero_pads_minutes",
      "args": ["--seconds", "65"],
      "stdin": "",
      "exit_code": 0,
      "stdout_contains": ["1:05"],
      "stderr_empty": true,
      "timeout_seconds": 5
    },
    {
      "id": "rejects_negative",
      "args": ["--seconds", "-5"],
      "exit_code": 2,
      "stderr_contains": ["must be non-negative"]
    },
    {
      "id": "missing_duration_arg",
      "args": [],
      "exit_code": 2,
      "stderr_contains": ["usage:"]
    }
  ]
}

The contract should come from observed behavior or a written specification, not from the model's explanation of its own patch. If you derive the expected behavior from the model, you are grading the model against its own guess and the gate loses its value.

Run it in three commands

First, create the patch from a tracked file. If the change includes untracked files, stage them or produce the patch in a way your review tooling already understands.

git diff HEAD -- scripts/format_duration.py > candidate.patch

Next, run the same contract against an unpatched worktree to make sure the baseline passes. If the baseline fails, fix the contract first; otherwise every candidate will inherit a red herring.

python contract_harness.py --repo . --contract contracts/format_duration.json

Finally, apply the candidate patch in the temporary worktree and run the cases again.

python contract_harness.py --repo . --patch candidate.patch --contract contracts/format_duration.json

A failing run prints the reason inline:

PASS zero_pads_minutes 0.11
PASS rejects_negative 0.09
FAIL missing_duration_arg 0.08
  reason: exit code 0 != 2; missing stderr fragment: 'usage:'
2/3 contract cases passed

Use a decision table instead of vibes

The harness should end in one of three buckets:

Result	What it tells you	Next step
All cases pass	The patch preserves the recorded CLI behavior.	Allow human review to focus on style, security, and edge cases outside the contract.
One or more cases fail	The patch changes an observable behavior.	Send the failing case back to the model, or drop the candidate.
The patch does not apply cleanly	The diff is stale, wrong base, or outside tracked files.	Regenerate the patch from HEAD before any model or review work.

Keep the output in a short evidence log. A single line such as {commit}: 3/3 contract cases passed is enough to make the review decision auditable later. Do not expand the contract until the current behavior is actually specified; adding dozens of examples after the fact creates a second test suite that drifts from the code.

Limitations and who should not use this

A contract harness is a sample, not a proof. It catches only the behaviors you wrote down, and it requires a stable executable target with deterministic text output. It is a poor fit for UI changes, async systems, database migrations, or security-sensitive code where a passing sample says little about risk. For those areas, keep your existing gates and treat the contract as a supplement, not a replacement.

The harness also assumes you can express a patch against HEAD and that the target command runs without network or secret dependencies. If your command needs a live database, credentials, or a long-lived process, the contract cases will be flaky and the gate will lose trust. The free server option is best used for controlled, non-sensitive examples; do not send private code or secrets to an endpoint unless your security review allows it.

Finally, do not mistake a passing contract for model agreement or correctness in general. The harness answers one narrow question: did this patch change an observable behavior that matters? If the answer is no, you still need a human to judge whether the new behavior is the right design.

If your team already has access to MonkeyCode's free model access and free server option, run this harness against one real candidate patch this week. The value is not the script itself; it is the evidence log you keep instead of arguing over whether a diff looks safe.