Opinion: Make the Free Model Write the Failing Test Before the Fix

Ask the model for the exam before the answer, then run that exam in three phases on infrastructure that costs nothing. A patch that cannot fail its own test is not a fix; it is a guess with formatting. Free model access changes the economics of generation, not the economics of trust, so the verification loop must become the product.

Why the Existing Tests Are the Wrong Oracle

Repository tests are a compromised oracle because the model has already seen them during training or inside the prompt context. A patch can therefore satisfy every existing assertion while violating the contract hidden in the issue text. The missing artifact is a fresh test that encodes the contract from the ticket alone, written before the model sees any implementation. That test becomes the falsification instrument: it fails on current code, passes on the patched code, and fails again when you break the patch deliberately.

This is not the same discipline as killing weak tests, which prunes a suite; this generates one strong test before the patch exists. The distinction matters because the test is the only artifact you can grade with certainty. A diff can look plausible, but a test either fails or it does not.

The Three-Phase Falsification Loop

The method assumes a ticket with a real behavioral contract and a boundary you can state in one sentence. Take a concrete example: a pricing module where loyalty customers currently receive 15% off, and the ticket says they receive an extra 5% when the subtotal is strictly above 100.

Step 1: Write the contract as a single sentence.

Reduce the issue to one behavioral statement with an explicit boundary, because models routinely get strict versus non-strict comparisons wrong. The boundary is the part that matters, and it belongs in the test before it belongs in the code.

Step 2: Ask the model for the test only.

You are writing a test, not a patch. Given this contract: "Loyalty
customers receive an extra 5% discount when the subtotal is strictly
above 100." Write a pytest test that fails on the current code if the
contract is not implemented. Do not read the repository and do not
propose an implementation. Include boundary cases.

Run this prompt before you paste any code into the context, because the moment the model sees the implementation it will write a test that matches the code instead of the contract. The model may return something close to this:

import pytest
from store.pricing import calculate_discount

def test_loyalty_extra_discount_above_threshold():
    assert calculate_discount(150, "loyalty") == pytest.approx(30.0)
    assert calculate_discount(100, "loyalty") == pytest.approx(15.0)
    assert calculate_discount(50, "loyalty") == pytest.approx(7.5)

Step 3: Run the baseline phase on a free server.

./falsify.sh baseline

The test must fail for the right reason: the extra discount is missing, not because of an import error or a typo. A failing test that fails for the wrong reason is worthless, and this is the step most teams skip because they are in a hurry to see green.

Step 4: Apply the model patch and run the patched phase.

./falsify.sh patched

If the test passes, you have evidence the patch satisfies the contract as written, which is necessary but not sufficient. The patch might still break an adjacent behavior that your single test does not cover.

Step 5: Mutate the patch and confirm the test catches it.

./falsify.sh mutation

The mutation phase is the actual opinion of this article: a test that cannot distinguish a correct patch from a broken one is decoration. Flip the comparison, delete the boundary check, or invert the tier condition, and the test should fail every time.

The script that runs all three phases is short enough to live in any repo:

#!/usr/bin/env bash
# falsify.sh — three-phase contract check against a model patch
set -euo pipefail

PHASE="${1:?usage: falsify.sh baseline|patched|mutation}"

case "$PHASE" in
  baseline)
    git checkout -- store/pricing.py
    pytest tests/test_contract.py -q
    ;;
  patched)
    git apply /tmp/model.patch
    pytest tests/test_contract.py -q
    ;;
  mutation)
    git apply /tmp/model.patch
    sed -i 's/price > 100/price >= 100/' store/pricing.py
    pytest tests/test_contract.py -q
    ;;
esac

The expected outcomes form a small decision table:

Phase	Action	Expected result
baseline	Run the test on current code	FAIL, because the contract is missing
patched	Apply the model patch, run the test	PASS, because the contract is satisfied
mutation	Break the boundary, run the test	FAIL, because the test catches the break

If the mutation phase passes, the test is too weak; send the test back to the model with the mutation diff and ask it to strengthen the assertions. This loop is the whole method, and it costs only the free model calls and the free server minutes.

Why Free Resources Fit This Loop

Free model access makes this loop practical because you can afford to regenerate the test several times without watching a meter, and a free server makes the three phases practical because you are not spending your team's CI budget on a test that is supposed to fail. MonkeyCode's free model access and free server option are enough to run this loop end to end, which is why the workflow is reproducible without a paid budget. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The combination turns the cheapest resources into the most valuable artifact: a falsifiable contract.

Limitations and Who Should Skip This

This method assumes the issue text contains a real behavioral contract, which is false for vague tickets like "improve performance" or "clean up this module". It also assumes the model's test does not encode the same misunderstanding as its patch, and when the issue is ambiguous both will be confidently wrong in the same direction. The mutation list is finite, so a test can pass all three phases and still miss a semantic regression that your chosen mutations did not cover.

Teams without a disposable server should run the same loop locally, because the phases are the method and the server is only a convenience. Do not use this workflow for large migrations or cross-cutting refactors, where a single contract test cannot capture the blast radius, and do not treat it as a substitute for human review of the diff. The test proves behavior at the boundary you chose, not the behavior the users actually need.

The Position

Free AI compute is best spent on the exam, not the answer, and the free server is the grading room where the exam is allowed to fail. The model will happily write your patch, but you should make it write the failing test first, because that is the only artifact you can actually grade. If you want to run this loop today, the free tier described above is sufficient; the script in this article is the entire harness.