Same-Prompt Green Is Not a Gate: Five Eval Myths

Who wrote the tests that just passed today?

I keep seeing the same broken eval loop. A model drafts the feature in one burst. Then it drafts a matching test file. Then both files go green on disk. Teams still call that a finished gate.

Is that a real gate, or only a mirror? This is an eval problem, not a prompt trick.

The claim I keep hearing

"The tests passed, so the feature works."

That sentence hides authorship on purpose. Who actually wrote the assertions? If the generator wrote them, they encode its reading. A wrong reading still yields a green bar.

I now use one boring split. Generation and evaluation never share a write path.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I sometimes draft an implementation with MonkeyCode's free model access. I run frozen checks on its free server option. The gate is the oracle script, not the chat transcript. Remove the product and the method still holds.

Myth 1: Passing tests mean the feature works

They mean those exact tests passed today. That is the whole fact.

If the model wrote the tests, they follow its story. The story can be wrong and still compile. Green then means agreement with itself.

Corrected mental model: a gate needs an oracle. The generator cannot edit that oracle.

Three questions before merge

• Who authored the test file on disk?
• Can the generator overwrite that path?
• Would a wrong implementation still pass?

No clear answers means you have no gate.

Myth 2: Coverage percent equals shipping confidence

Coverage is a map of executed lines. It is not a verdict.

Generated tests love happy paths and friendly mocks. They assert on stubs they just created. High coverage of a tautology remains a tautology.

I do not treat coverage as a ship signal. I treat it as a hole finder only.

What I record instead

• Independent oracle pass or fail
• Fixture source: human-owned and frozen
• One mutated requirement, then a re-run

If flipping a requirement leaves tests green, those tests are decoration.

Myth 3: Asking the model to verify is review

You paste the code back into chat. You type verify like a second reviewer.

The same weights then reread the same story. That is not independence. A second sample is not a second specification.

Corrected mental model: verification is a program with an exit code. Use a script, a fixture, and a locked path.

# proposed layout — template, not a measured run
oracle/
  test_contract.py    # frozen; generator cannot write here
  fixtures/
    cases.json        # human-owned examples
src/
  impl.py             # generator may write only here
scripts/
  run_oracle.sh

Keep oracle/ out of the generate step. Keep src/ as the only writable target. If both paths are open, the split is theater.

Myth 4: A green free-server run is already CI

A free server is one convenient machine. It is not your pipeline.

The chat said pass. The server printed pass. That still is not ship.

What binary ran, and against which tree? Which test path actually executed on that host? Did the job mount the frozen oracle directory? Or did it run tests the model just emitted?

I want the command in the log. I want the oracle path. I want the process exit code.

# proposed runner — treat as unexecuted on your box
set -euo pipefail
ROOT="$(pwd)"
test -f "$ROOT/oracle/test_contract.py"
test -f "$ROOT/oracle/fixtures/cases.json"

# Fail closed if the generator can still rewrite the judge.
if [ -w "$ROOT/oracle/test_contract.py" ]; then
  echo "oracle_writable=1" >&2
  echo "Lock oracle/ before generation. Refusing to score." >&2
  exit 2
fi

python -m pytest "$ROOT/oracle/test_contract.py" -q
echo "oracle_exit=$?"

Make the oracle read-only before generation. Then generate into src/ only. Then run the script above.

If the model "fixes" tests to stay green, your gate died. A convenient host does not restore independence.

Myth 5: Model-made snapshots catch real regressions

Snapshots record what happened in one run. They do not know what should happen.

A model can snapshot its own bug. Next week that bug is golden. The suite then protects the defect.

I freeze expected values before any implementation exists. That JSON file is the spec. The code remains the suspect.

Tiny oracle example

# oracle/test_contract.py
# Proposed example. Not a measured benchmark.
import json
from pathlib import Path

from src.impl import normalize_login

CASES = json.loads(Path("oracle/fixtures/cases.json").read_text())


def test_emails_are_lowercased():
    for row in CASES["emails"]:
        assert normalize_login(row["in"]) == row["out"]


def test_rejects_empty():
    for bad in CASES["empty"]:
        try:
            normalize_login(bad)
        except ValueError:
            continue
        raise AssertionError(f"accepted empty: {bad!r}")

{
  "emails": [
    {"in": "[e-mail]", "out": "[e-mail]"},
    {"in": "  [e-mail]  ", "out": "[e-mail]"}
  ],
  "empty": ["", "   "]
}

# src/impl.py — generator may rewrite this file only
def normalize_login(value: str) -> str:
    text = value.strip().lower()
    if not text:
        raise ValueError("empty login")
    return text

The model may write src/impl.py. It may not touch oracle/. That split is the whole method.

The workflow I actually use

Not a product tour. A sequence you can copy.

1. Write the contract first. JSON cases. Two failing tests.
2. Lock the oracle directory. Generator write path is src/ only.
3. Draft impl.py with a free model. One function. No test files.
4. Run scripts/run_oracle.sh on a free server or locally.
5. Read the exit code. Ignore the chat summary.
6. If it fails, change code. Not cases. Not tests.

Need a new requirement? You edit the oracle. You do that by hand.

Decision table

Signal	Treat as	Do not treat as
Chat says "all tests passed"	A claim	A gate
Tests generated in the same prompt	Notes	Evidence
Coverage from those tests	A map	Confidence
Frozen oracle exit 0	A gate	Product-market proof
Host log with the pytest command	A run record	Production CI

Print the table next to the pull request. Ask which cell you are using. Most fights end there.

A mutation check you can finish quickly

Do this once per feature. It stays cheap.

1. Run the oracle. Note the exit code.
2. Break one branch in impl.py on purpose.
3. Re-run. Expect a failure.
4. Restore the code to the prior tree.
5. Break one row in cases.json on purpose.
6. Re-run. Expect a failure.

If step 3 stays green, tests never saw the behavior. If step 6 stays green, tests never saw the spec. That is the whole point of independence.

# proposed mutation smoke — still a template
python -m pytest oracle/test_contract.py -q
# sabotage impl, re-run, expect non-zero
# restore impl
# sabotage one cases.json row, re-run, expect non-zero
# restore cases.json

I also keep a tiny deny list in review. No new test_*.py from the generator. No edits under oracle/. No "updated snapshots" in the same change as impl.py.

# proposed review grep
git diff --name-only origin/main...HEAD | grep -E '^(oracle/|test_)' && echo reject

If that grep prints a path, stop. The judge moved with the suspect.

What a chat summary is worth

A chat summary is narration. It is not an artifact.

"All tests passed" has no tree hash. It has no command. It has no locked oracle path. I do not paste that sentence into a merge box.

I paste the command, the exit code, and the file list. That bundle can be replayed. The sentence cannot.

Limitations

This does not replace code review. It does not replace staging.

It does not prove performance. It does not prove security. A frozen oracle is only as good as the cases you wrote.

Bad cases. Green bar. Same lie, smaller room.

Free model access will not invent a missing requirement. A free server will not make a circular test independent. If your contract lives in chat history, you do not have a contract.

Flaky tests still flake under this layout. Locking the path does not lock the network. Keep I/O out of the oracle unless the contract is I/O.

Who should not use this

Do not use this as your only check for safety-critical code. Do not freeze tests you do not understand.

Do not point the generator at oracle/ "just this once." Do not skip human cases because the model offered more.

If you cannot write two examples by hand, you are not ready to generate the implementation. If your team cannot keep a directory read-only, this method will rot.

Corrected mental model

The model is a proposer. The oracle is a judge.

They cannot be the same process. Green means the judge agreed. It does not mean the judge is wise.

Write the judge first. Keep it small. Keep it frozen. Then let a model try.

If you already have free model access and a free server option, try this layout on one function. Keep the chat out of the gate.