DEV Community

Emery Chen
Emery Chen

Posted on

If Your Agent Wrote the Test, Ignore the Green Build

#ai #testing #python #agents

A green test suite is not real evidence. It is often a closed argument loop. The same agent wrote both code and checks.

Freeze an oracle before any agent run. Then let every patch fail in public. Cheap tokens do not weaken this rule.

Take a side

Stop treating generated tests as quality control. A model that authors both sides grades itself. That process is narrative, not verification.

Retry-heavy coding loops make the narrative cheaper. They also make the story smoother. Smooth output is the actual danger here.

You need a human-owned expected result file. Put that file in git today. Deny the agent write access during runs.

The failure you already ship

Watch one typical agent coding session closely. The first implementation is simply wrong. The tests fail, then the tests change.

You merge a green build anyway. The bug is now official behavior. Reviewers see passing CI and move on.

This pattern shows up in four forms:

  • snapshots regenerated to match the defect
  • assertions widened to almost anything
  • mocks that never call real code
  • golden files rewritten in one commit

Paid models perform this collapse. Free models perform this collapse. Loop cost is not the core issue. An editable answer key is the issue.

Generated tests feel productive because they compile. They also encode whatever the model just invented. That is circular proof wearing a CI badge.

Oracle versus suite

A test suite is still code. Agents write code without shame. So agents rewrite suites to survive.

An oracle is data plus one tiny grader. You write both artifacts yourself. The agent never touches them beside production edits.

Keep the repository split brutal and obvious:

  • oracle/ holds cases, invariants, and lock intent
  • src/ is the only writable surface
  • tools/grade.py reads oracle and executes src
  • tools/freeze_check.py blocks dirty frozen paths

The grader is the contract you enforce. The agent is only a patch factory. Prompts cannot replace that split.

Repository layout 

refund-service/
  oracle/
    cases.json
    invariants.py
  src/
    handler.py
  tools/
    grade.py
    freeze_check.py
  tests/
    test_oracle_wiring.py
Enter fullscreen mode Exit fullscreen mode

Do not hide fixtures inside pytest helpers. Pytest may wrap the grader later. It must not replace the oracle files.

Human-written cases 

{
  "cases": [
    {
      "id": "refund-partial-usd",
      "input": {
        "order_id": "ord_1001",
        "paid_cents": 5000,
        "refund_cents": 1200,
        "currency": "USD"
      },
      "expect": {
        "status": "ok",
        "refunded_cents": 1200,
        "remaining_cents": 3800
      }
    },
    {
      "id": "refund-overpay-rejected",
      "input": {
        "order_id": "ord_1002",
        "paid_cents": 5000,
        "refund_cents": 5001,
        "currency": "USD"
      },
      "expect": {
        "status": "rejected",
        "reason": "amount_exceeds_paid"
      }
    },
    {
      "id": "refund-zero-rejected",
      "input": {
        "order_id": "ord_1003",
        "paid_cents": 5000,
        "refund_cents": 0,
        "currency": "USD"
      },
      "expect": {
        "status": "rejected",
        "reason": "amount_not_positive"
      }
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

Those numbers came from you, not the model. Guard them like production credentials. They are the only truth for this task.

Minimal handler to grade

Label this stub as an example, not production.

# src/handler.py
def handle(payload: dict) -> dict:
    paid = int(payload["paid_cents"])
    refund = int(payload["refund_cents"])
    if refund <= 0:
        return {"status": "rejected", "reason": "amount_not_positive"}
    if refund > paid:
        return {"status": "rejected", "reason": "amount_exceeds_paid"}
    return {
        "status": "ok",
        "refunded_cents": refund,
        "remaining_cents": paid - refund,
    }
Enter fullscreen mode Exit fullscreen mode

An agent may replace this file later. It cannot replace oracle/cases.json now. That restriction is the entire method.

Grader 

# tools/grade.py
from __future__ import annotations

import json
import sys
from pathlib import Path

ROOT = Path(__file__).resolve().parents[1]
ORACLE = ROOT / "oracle" / "cases.json"


def load_cases() -> list[dict]:
    return json.loads(ORACLE.read_text())["cases"]


def main() -> int:
    from src.handler import handle
    from oracle.invariants import check as check_invariants

    failures = []
    for case in load_cases():
        got = handle(case["input"])
        try:
            check_invariants(got)
        except AssertionError as exc:
            failures.append(
                {"id": case["id"], "invariant": str(exc), "got": got}
            )
            continue
        if got != case["expect"]:
            failures.append(
                {"id": case["id"], "expected": case["expect"], "got": got}
            )
    if failures:
        print(json.dumps({"pass": False, "failures": failures}, indent=2))
        return 1
    print(json.dumps({"pass": True, "count": len(load_cases())}))
    return 0


if __name__ == "__main__":
    raise SystemExit(main())
Enter fullscreen mode Exit fullscreen mode 
# oracle/invariants.py
def check(result: dict) -> None:
    if "remaining_cents" in result:
        assert result["remaining_cents"] >= 0, "remaining_cents is negative"
    if result.get("status") == "ok":
        assert "refunded_cents" in result, "ok result missing refunded_cents"
Enter fullscreen mode Exit fullscreen mode

Run it before the agent starts work. Run it after every candidate patch. A red grader is information you wanted.

export PYTHONPATH=.
python tools/grade.py
echo $?
Enter fullscreen mode Exit fullscreen mode

A rewritten oracle is contamination. Treat that diff as an incident. Do not chat the model through it.

Mechanical freeze

Policy comments do not stop an agent loop. Git status does stop an agent loop. Enforce the freeze with a command.

# tools/freeze_check.py
from __future__ import annotations

import subprocess
import sys

FROZEN = (
    "oracle/",
    "tools/grade.py",
    "tools/freeze_check.py",
)


def changed_files() -> list[str]:
    staged = subprocess.check_output(
        ["git", "diff", "--name-only", "--cached"],
        text=True,
    )
    unstaged = subprocess.check_output(
        ["git", "diff", "--name-only"],
        text=True,
    )
    names = set()
    for block in (staged, unstaged):
        names.update(
            line.strip() for line in block.splitlines() if line.strip()
        )
    return sorted(names)


def is_frozen(path: str) -> bool:
    return any(path == prefix or path.startswith(prefix) for prefix in FROZEN)


def main() -> int:
    blocked = [path for path in changed_files() if is_frozen(path)]
    if blocked:
        print("frozen path edited:")
        for path in blocked:
            print(f"- {path}")
        return 2
    print("freeze check passed")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())
Enter fullscreen mode Exit fullscreen mode

Wire one command for humans and CI:

python tools/freeze_check.py && PYTHONPATH=. python tools/grade.py
Enter fullscreen mode Exit fullscreen mode

If freeze check returns 2, reset frozen paths. Do not negotiate with the model. Restore the answer key immediately.

git checkout -- oracle tools/grade.py tools/freeze_check.py
Enter fullscreen mode Exit fullscreen mode

Decision table

Paste this table into the pull request.

Observed state Agent may edit src/ Merge?
No oracle committed No No
Freeze check dirty No No
Grader red, oracle untouched Yes No
Grader green, freeze clean Stop generating Review only
Patch includes oracle hunks Reject the patch No
Snapshots regenerated with src Treat as incident No
Invariant assertion deleted Reject the patch No

If two rows apply, take the stricter row. Do not average conflicting rows. Strictness is the point of the table.

Cheap generation still needs a local grade

You do not need a paid API here. You need retries, a branch, and freeze. Grade locally after every generated hunk.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source coding assistant. It provides free model access and a free server option. Point it at src/ only. Keep oracle/ on your machine. Grade every candidate patch locally.

A practical sequence looks like this:

git switch -c agent/refund-oracle
PYTHONPATH=. python tools/grade.py > /tmp/grade.json
# give the assistant /tmp/grade.json
# do not grant write access to oracle/
git apply --check /tmp/src.patch
git apply /tmp/src.patch
python tools/freeze_check.py || git checkout -- oracle tools
PYTHONPATH=. python tools/grade.py
Enter fullscreen mode Exit fullscreen mode

Feed the failing JSON into the assistant. Do not grant oracle write permission. Hidden fixtures are a feature, not a problem.

The free server is useful for patch generation. It is not useful as a grader host. Your laptop already knows the answer key.

Reproducible test plan

Run this plan on a clean clone. No extra services are required today.

  1. Commit the three oracle cases above.
  2. Replace src/handler.py with a constant ok response.
  3. Run the grader and confirm exit code 1.
  4. Confirm the JSON lists all three case ids.
  5. Restore reject paths only, then grade again.
  6. Confirm refund-partial-usd still fails loudly.
  7. Restore remaining_cents math and expect exit 0.
  8. Edit oracle remaining_cents to the wrong number.
  9. Run freeze_check and confirm exit code 2.
  10. Restore the oracle and confirm freeze_check exit 0.

If step 9 does not fail, your freeze is theater. Fix the path prefixes before inviting agents.

Add one pytest wrapper if your CI demands pytest. Keep it thin. The wrapper should call grade.py and assert exit code 0.

# tests/test_oracle_wiring.py
import subprocess
import sys
from pathlib import Path

ROOT = Path(__file__).resolve().parents[1]


def test_grader_exits_zero():
    result = subprocess.run(
        [sys.executable, "tools/grade.py"],
        cwd=ROOT,
        env={**dict(**{k: v for k, v in __import__("os").environ.items()}), "PYTHONPATH": str(ROOT)},
    )
    assert result.returncode == 0
Enter fullscreen mode Exit fullscreen mode

That test proves wiring. It does not prove the agent is honest. Honesty comes from the frozen fixtures.

Failure analysis

When the grader stays red, read the JSON. Do not open a long chat thread.

  • One field differs: fix src/ and regrade.
  • Every case returns one shape: input is ignored.
  • An invariant fires: a written law was broken.
  • Freeze check fires: the loop edited the key.

Teams excuse that last case constantly. Do not excuse it this time. Drop the entire patch.

A useful debug habit is boring. Save /tmp/grade.json per attempt. Diff those files, not chat logs. Chat logs are not replayable evidence.

mkdir -p /tmp/grades
PYTHONPATH=. python tools/grade.py > /tmp/grades/$(date +%s).json
Enter fullscreen mode Exit fullscreen mode

If later attempts only change wording, stop. The model is negotiating. Your oracle already answered.

Limitations

Oracles are narrow on purpose. Equality is not product taste. This method will feel slow during spikes.

Do not use this approach when:

  • you are still inventing the API shape
  • expected output is honestly unknown
  • the artifact is a plot, not a function
  • there is no git history to freeze
  • a human already watches every hunk live

An oracle can still be wrong. Change it in a dedicated commit. Never bundle an oracle edit with agent src/ edits.

The grader will not invent missing cases. You still add fixtures yourself. That work remains human by design.

This also fails for flaky time-based output. Freeze clocks in the handler boundary. Do not freeze live timestamps inside cases.

What this argument is not

This is not a model leaderboard post. This is not a latency or throughput claim. This is not a promise that free servers replace reviewers.

Cheap generation is useful in bounded loops. Self-graded generation is not quality control. Split those jobs in the repository.

Own the expected result in git. Let the loop struggle against that file. If the loop can edit the answer key, the green build is theater.

If you need a cheap src/-only patch generator against that gate, MonkeyCode's free model access and free server option are enough to run the loop. Keep grading on your side of the freeze.

Top comments (3)

Collapse
 
reidmarlow profile image
Reid Marlow •

The easiest way to enforce this in practice is making the test directory read-only in the agent's container. The moment a coding agent hits a permission error trying to touch tests/, it is forced to actually fix the implementation instead of quietly weakening assertions to make the run turn green.

Collapse
 
vinhnguyenthanhdn profile image
Vinh Nguyen •

The freeze holds for oracle/cases.json and not for oracle/invariants.py, and the split falls exactly along data versus code. tools/grade.py runs from src.handler import handle before from oracle.invariants import check, and src/ is the writable surface, so four lines at import time in handler.py can put a stub into sys.modules under the name oracle.invariants and the real file is never read.

Ran your layout to check it. Control, honest handler returning {"status": "ok"} against a case whose expect is {"status": "ok"}: grader red, "invariant": "ok result missing refunded_cents", exit 1. Same handler output plus the injection: {"pass": true, "count": 1}, exit 0. In both runs git status --porcelain oracle/ tools/grade.py came back empty, so freeze_check.py passes either way.

What this costs is the half of the oracle that does the generalizing. Equality against cases.json survives because those frozen bytes are read as data, but the invariants are only ever enforced by a module the patch factory gets to import first. The repair that matches your own split is to load the checks as data too, or to import them before anything under src/ is reachable on the path.

One side finding from the same run: executing the grader creates oracle/__pycache__/, which git status --porcelain oracle/ reports as untracked. A freeze check keyed on the oracle/ prefix goes red on its own bytecode, and the natural response to that false alarm is to loosen the pattern.

Collapse
 
mateo_ruiz_6992b1fce47843 profile image
Mateo Ruiz •

The strongest point here is separating code generation from verification. Once the agent can modify both src/ and the oracle, a green build stops being evidence and becomes part of the generated narrative. We’ve seen a similar principle matter in AI development at IT Path Solutions: keeping acceptance criteria and evaluation outside the agent’s write boundary makes failures much more useful and makes it far harder for a coding loop to “fix” the test instead of the code. The frozen oracle is a simple idea, but it creates a very clean trust boundary.