Free Models Are Bad Reviewers but Useful Test Factories

Why this is worth reading: if you paste a diff into a free model and ask “is this safe?”, you get an opinion you cannot verify. If you instead ask the same model to write a failing test that captures the behavior you are changing, you get an artifact you can run. This article gives you a loop that turns a free model endpoint into a test generator, executes the generated tests in a temporary git worktree, and fails the run on suspicious imports or network calls. You keep the useful part of the model — speed and cheap coverage suggestions — and throw away the part you should not trust: its confidence.

The operator behind MonkeyCode reports that it offers free model access and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The example script uses MONKEYCODE_ENDPOINT and MONKEYCODE_API_KEY because they are already available in the environment; you can substitute any OpenAI-compatible endpoint.

The shift that makes verification possible

A model that reviews a patch gives you a probability, not a proof. Even when it says “this looks correct,” you cannot distinguish a useful check from a fluent hallucination. A model that generates a test gives you code. If the code is wrong, it either fails to run or fails to exercise the intended behavior. Both outcomes are observable in CI.

You still should not trust the generated test blindly, but you have a much better failure mode. Instead of arguing with a model about whether a patch is safe, you run a generated test against the patched code. The test either passes, fails, or does not run. Each result tells you something concrete.

The workflow below assumes you have a Python repository with pytest in your test dependencies. If your stack is different, the pattern transfers: keep the generator outside your repo, run its output inside a temporary worktree or container, and add one automated safety check before execution.

A generator you can run today

Save this as generate_and_run_tests.py:

#!/usr/bin/env python3
"""Generate property tests from a diff and run them in a temporary worktree."""
import argparse
import os
import subprocess
import sys
import tempfile
from pathlib import Path

import httpx


ASSISTANT_SYSTEM = (
    "You are a test generator. Read the diff and emit only a self-contained "
    "pytest file. Use Hypothesis property-based tests where possible. Import "
    "only modules that already exist in the repository. Do not include network "
    "calls, subprocess, file system writes outside the test's own tmp_path, "
    "or infinite loops. The first line of your output must be exactly "
    "'# generated-test'."
)

FORBIDDEN_IMPORTS = {
    "subprocess",
    "socket",
    "requests",
    "urllib",
    "os.system",
}


def call_model(diff: str) -> str:
    response = httpx.post(
        f"{os.environ['MONKEYCODE_ENDPOINT']}/chat/completions",
        headers={"Authorization": f"Bearer {os.environ['MONKEYCODE_API_KEY']}"},
        json={
            "messages": [
                {"role": "system", "content": ASSISTANT_SYSTEM},
                {"role": "user", "content": diff},
            ]
        },
        timeout=30,
    )
    response.raise_for_status()
    return response.json()["choices"][0]["message"]["content"]


def extract_code_block(text: str) -> str:
    if "```

python" in text:
        return text.split("

```python", 1)[1].split("```

", 1)[0]
    if "

```" in text:
        return text.split("```

", 1)[1].split("

```", 1)[0]
    return text


def check_forbidden_imports(code: str) -> None:
    for line in code.splitlines():
        stripped = line.strip()
        if stripped.startswith(("import ", "from ")):
            for forbidden in FORBIDDEN_IMPORTS:
                if forbidden in stripped:
                    raise SystemExit(f"forbidden import in generated test: {stripped}")


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("diff_file", help="path to a unified diff")
    args = parser.parse_args()

    diff_text = Path(args.diff_file).read_text()
    generated = extract_code_block(call_model(diff_text))
    if not generated.startswith("# generated-test"):
        raise SystemExit("generated test did not include the required marker")

    check_forbidden_imports(generated)

    with tempfile.TemporaryDirectory() as tmpdir:
        worktree = Path(tmpdir) / "repo"
        subprocess.run(
            ["git", "worktree", "add", "--detach", str(worktree), "HEAD"],
            check=True,
            capture_output=True,
        )
        test_path = worktree / "tests" / "test_generated.py"
        test_path.parent.mkdir(parents=True, exist_ok=True)
        test_path.write_text(generated)

        result = subprocess.run(
            ["pytest", "-q", "tests/test_generated.py"],
            cwd=worktree,
        )
        return result.returncode


if __name__ == "__main__":
    raise SystemExit(main())

Run it from your repository root:

git diff origin/main > /tmp/change.diff
python generate_and_run_tests.py /tmp/change.diff

The script is deliberately strict about the marker and forbidden imports. If the model wraps the code in a paragraph instead of returning only the file, the script fails. If it tries to import requests, the script fails. Both failures happen before any generated code runs.

Why the temporary worktree matters

A generated test can contain a destructive command, a call that reads environment variables, or a pytest fixture that mutates shared state. Running it in a detached worktree keeps most damage confined to a copy of the repository, not your working tree. The worktree also prevents the generator from overwriting your existing tests or source files.

You could go further and run the generated test inside a container with no network and a read-only root filesystem. For most small repositories, a worktree is the right trade-off between safety and setup cost. If your code handles secrets or production data, use a container.

What can still go wrong

The most common failure is not a malicious generated test. It is a generated test that always passes. A model can write def test_ok(): assert True, satisfy the prompt, and tell you nothing. To catch that, check that the generated file contains at least one import from your repository and at least one non-trivial assertion. A simple grep is not enough because the model can fabricate imports, but it filters the laziest cases.

A second failure is a test that relies on an internal function that does not exist. The generator may hallucinate a module name or a function signature. The CI run will fail with an import error, which is a correct outcome, but it costs you a cycle. You can reduce this by passing the diff plus a short tree of relevant files to the model, but do not pass the entire repository if it contains secrets.

Finally, the model may generate a test that passes for the wrong reason. This is the hardest case and the reason this workflow never replaces human review. It only moves the verification step from “does the model think it is correct?” to “does a machine-checkable artifact fail?” The latter is stronger, but not complete.

Who should not use this approach

If your repository has no test framework or no stable test command, generated tests will not run cleanly. You will spend more time fixing harness issues than getting signal. If the patch changes SQL migrations, infrastructure, or shell scripts, a generated pytest file is likely the wrong artifact. Use a migration-specific or shell-specific gate instead.

Also skip this if your free endpoint has a very low rate limit and you plan to generate tests on every commit. Run the generator on demand, or cache a generated test suite and compare it against the diff. The free tier becomes a review aid, not a dependency.

If you already have a free model endpoint idle in your CI, take one small diff and ask it to generate a failing test rather than an opinion. You will learn more from the test run than from another confident paragraph.