Your Model Graded Its Own Homework

A green log is not a finished patch. I start from that rule every single time. If the generator also wrote the test, you graded nothing.

I keep a short catalog for free model drafts. Each entry lists a symptom, a cause, and a replacement. I do not need a new slogan to spot the miss.

What I refuse to trust

Free model access can draft a unified diff. A free server can run that diff one time. Neither option is allowed to define the expected behavior.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Do you want a draft, or do you want a verdict? I want the draft, and I keep the verdict. I use those two options as a rehearsal bench only.

I do not treat a chat reply as a specification. I do not treat a remote exit code as proof. A friendly log can still hide a fake oracle.

Anti-pattern 1: the model writes the oracle

Symptom

The new tests mention the same names as the patch. They pass on the first remote run you try. You feel finished, and that feeling is the trap.

Root cause

One generator filled both sides of the check. The test encodes the implementation you just accepted. It cannot catch a wrong idea with a pretty shape.

Replacement

I write two fixture files before I ask for any code. One file holds the input object I care about. The other file holds the output I will accept.

The model may not edit either fixture file. Could a stranger fail this test without your patch? If the answer is no, your oracle is fake.

Write a stricter fixture, then ask the model again. Then rerun the compare before you trust the color. These commands are an unexecuted example, not a lab report.

mkdir -p rehearsal/contract
printf '%s\n' '{"n": 4}' > rehearsal/contract/input.json
printf '%s\n' '{"ok": true, "double": 8}' > rehearsal/contract/expect.json
test -s rehearsal/contract/expect.json && echo "fixture ready"

Change the expected numbers to match your real rule. Do not let the model pick them after it sees the diff. Pair the files with a short note you wrote yourself.

# Behavior note

Input n is an integer from 0 to 20.
Return ok true only when n is even.
Return double as n times 2 when ok, else 0.

That note is the spec the model is allowed to see. If the note and the fixture disagree, fix the note first. Do not ask the model to reconcile your own contradiction.

Anti-pattern 2: fixtures arrive after the diff

Symptom

The patch lands in the branch before any example. The sample numbers show up later in the same chat. Those numbers match the code you already wanted.

Root cause

You let the result choose the requirement after the fact. Time order is the actual bug, not the syntax. A contract should be older than the candidate patch.

Replacement

I record file times before I send any prompt. The gate fails when the fixture is newer than the patch. That rule feels strict, and it should feel strict.

stat -c '%Y %n' rehearsal/contract/expect.json rehearsal/patch.diff

On macOS I switch to stat -f '%m %N' for the same check. I label this snippet as a local proposal, not a benchmark. Which file did you create first, the fixture or the diff?

If you cannot tell, you do not have a contract yet. Stop the prompt and write the fixture in git first. A chat bubble is not a timestamp you can audit.

Anti-pattern 3: the free box becomes the spec

Symptom

The script runs on the free server and looks fine. The same script fails on your laptop five minutes later. The missing import was never written into your lockfile.

Root cause

An unnamed runtime quietly became your real design. Locale, timezone, and preinstalled packages hid the gap. You then copied host notes into the repo as requirements.

Replacement

I pin the command, the workdir, and the expected files. The remote host may execute the candidate one time. It may not invent requirements I did not write down.

Check	Free server may do	You must own
Run the candidate once	Yes	The command text
Print stdout and stderr	Yes	The fixture files
Add a missing package	No	Your lockfile
Decide pass or fail	No	Your local compare
Store a secret or token	No	Your redaction step

If a package exists only on that host, I stop. I add it locally, or I drop the idea entirely. Does your lockfile explain every import the log used?

Anti-pattern 4: you skip the deleted lines

Symptom

The viewer paints the new lines in a friendly green. A helper function vanished near the bottom of the file. The new test never called the path you just lost.

Root cause

Your eyes follow additions because additions look like work. Generated patches delete quietly while you read the headline. Addition bias is how bounds checks disappear unnoticed.

Replacement

I count removed lines before I read any new line. A deletion without a contract note is an automatic reject. I want a human sentence that explains each removal.

git diff --numstat -- candidate.py
git diff --unified=0 -- candidate.py | grep '^-' | grep -v '^---'

Why did that line leave the file at all? If you cannot answer in one plain sentence, stop. Leave the branch dirty until the note exists.

Anti-pattern 5: comments pretend to be a fix

Symptom

The diff is long, but behavior barely moved at all. New comments restate the old function in softer words. The free server still prints the same wrong value.

Root cause

Activity got mistaken for progress during a fast review. A verbose patch feels safer than a small sharp edit. The generator optimized for looking busy, not the fixture.

Replacement

I reject patches whose comment lines dominate the behavior lines. Docs belong in the notes file when I asked for docs. Otherwise I want a smaller diff that moves the output.

Did the output change, or did the narration change? Grade the fixture, not the paragraph under the function. A longer file is not a stronger fix.

Run this gate after the remote log comes home

This script is a proposal you can read today. I am not publishing timing numbers or success rates. It checks shape, notes, and a plain JSON compare.

It does not deploy, install, or call a network. Treat every command below as unexecuted sample code. Adapt paths if your repo layout differs from this example.

#!/usr/bin/env python3
"""Local contract gate. Proposal, not a measured benchmark."""
import json
import sys
from pathlib import Path

ROOT = Path("rehearsal")
NEED = ("input.json", "expect.json", "notes.md")

def fail(msg: str) -> None:
    print(f"REJECT {msg}")
    sys.exit(2)

def main() -> None:
    box = ROOT / "contract"
    if not box.is_dir():
        fail("missing contract dir")
    for name in NEED:
        if not (box / name).is_file():
            fail(f"missing {name}")
    expect = json.loads((box / "expect.json").read_text())
    got_path = ROOT / "server_out.json"
    if not got_path.is_file():
        fail("missing server output")
    got = json.loads(got_path.read_text())
    if got != expect:
        fail("output != fixture")
    notes = (box / "notes.md").read_text().strip()
    if len(notes) < 40:
        fail("notes too thin")
    print("PASS local contract compare")

if __name__ == "__main__":
    main()

Copy the remote stdout into rehearsal/server_out.json before the run. Then invoke the gate and read the exit code yourself. A non-zero exit means reject, not a warmer prompt.

python3 rehearsal/gate_patch.py
echo "exit=$?"

A pass means your output matched the fixture you wrote. A pass does not mean the patch is wise or safe. A pass does not mean the remote host was clean.

Preflight before the prompt

I run four checks before I ask for any draft.

1. Confirm both fixture files exist and are non-empty.
2. Confirm notes.md states the behavior in plain words.
3. Scan the prompt text for keys before you paste.
4. Send the contract, not your whole repository tree.

grep -nE '(AKIA|BEGIN PRIVATE KEY|password\s*=)' prompt.txt && echo "stop"

That sniff is a seatbelt, not a full secret scanner. If it prints stop, I rewrite the prompt without the secret. I never paste a live environment file to save time.

Where a free model and a free server fit

I ask for a patch against notes.md, not against a vibe. I send no secrets and no environment file with the ask. I run the candidate once on the free server option.

I bring stdout home and compare it on my machine. MonkeyCode is the bench this rehearsal is written around. Free model access drafts the patch from the notes I wrote.

The free server option runs that draft away from my laptop. I still own the contract, the fixture, and the reject. Remove the product name and the gate still stands.

I am not stating a token quota in this article. I am not naming a model identifier or a hardware size. I am not promising how long any free offer stays up.

Check the live offer before you plan a week around it. Terms move, and stale numbers become confident lies. A missing quota line is safer than a fabricated one.

Is a fake quota more helpful than an honest gap? No, because a fake number schedules work you cannot finish. I would rather send you to the current page.

Who should not use this approach

Skip this if you cannot read a diff line by line. Skip this if the change touches auth, payments, or private data. Skip this if you need a certified or reproducible release build.

A free remote host is a rehearsal room, not a release gate. Skip this if your fixture says only "make it better." That sentence is a mood, not a contract you can test.

Write the input, write the output, then ask for code. Teams with no human reviewer should not adopt this flow. The script can reject a mismatch, not a bad product idea.

Judgment stays with a person who can own the merge. If that person is absent, do not automate the yes. Wait until a reviewer can read the deletions.

What I keep on the next pass

I keep four questions on a card beside the keyboard.

1. Who wrote the oracle that this run claims to satisfy?
2. Which file is older, the fixture or the candidate patch?
3. What did the host add that my lockfile does not list?
4. Which lines disappeared, and where is the note?

If any answer is mush, I stop the merge right there. The draft can stay on the branch until the note exists. The verdict stays with me, not with the remote log.

If you already keep fixtures in git, rehearse this once. Use a free model pass only to draft against your notes. Use one free server run only to collect the remote log.

Then let the local script be the only grade you trust. That is the whole offer I am willing to make today. No countdown, no ranking claim, and no borrowed story.