DEV Community

Sam Chen
Sam Chen

Posted on

Green CI, Bad Patch: Five Oracle Anti-Patterns

#ai #testing #agents #python

Your agent did not succeed, and your oracle lied. I keep watching green checks land on broken diffs. The suite never asked the right question, did it?

This is not another agent-loop post from me. This post is about the judge you trust. If the judge is cheap, every model looks brilliant.

The point

Stop swapping endpoints. Fix the success check first.

I want a judge I can rerun tomorrow. Same input. Same fail. No vibes allowed.

What I mean by an oracle

An oracle decides this coding run worked. It is not the model. It is not the prompt either.

It is the gate you actually trust. Most teams skip this gate. Then they argue about models. Wrong fight, every single time.

How to read this catalog

I use three fields. Every time. No extra theater.

  • Symptom: what the log shows
  • Root cause: why the gate is lying
  • Replacement: what I run instead

Steal the replacements. Leave the excuses on the floor.

1. The exit-code oracle

Symptom: pytest returns zero. You merge.

Root cause: zero only means the process lived. It does not mean tests ran. Collection can fail open. Markers can drop every case.

Skips look like wins. Empty suites look like wins. That is the trap.

Replacement: assert collected, failed, and skipped counts. Empty collection is a red gate.

# proposed gate — unexecuted example
import json, subprocess

def collect_pytest():
    p = subprocess.run(
        ["pytest", "-q", "--collect-only", "-p", "no:cacheprovider"],
        capture_output=True, text=True,
    )
    return p.returncode, p.stdout
Enter fullscreen mode Exit fullscreen mode

Collection is lock one. I still run the suite.

def assert_pytest_really_ran(report_path="report.json"):
    data = json.loads(open(report_path).read())
    summary = data["summary"]
    collected = summary.get("collected", 0)
    skipped = summary.get("skipped", 0)
    failed = summary.get("failed", 0)
    if collected == 0:
        raise SystemExit("oracle-fail: zero tests collected")
    if skipped == collected:
        raise SystemExit("oracle-fail: every test skipped")
    if failed:
        raise SystemExit(f"oracle-fail: {failed} failed")
Enter fullscreen mode Exit fullscreen mode

Did your last green run collect anything real? Check the counts before you brag.

2. The snapshot the model wrote

Symptom: golden files match. Reviewers relax.

Root cause: the agent wrote the snapshot too. The test now certifies the bug. You froze the hallucination in git.

Replacement: freeze expected output from humans. The agent may read fixtures. It may not rewrite them in the same turn.

Who wrote the expected file Merge oracle?
Human, before the agent ran Yes
Agent, same turn as the patch No
Agent, later, after a failing test Only with a human ack

I keep fixtures in testdata/frozen/. That directory is read-only in the tool schema. Simple rule. Hard to dodge on purpose.

3. The compile oracle

Symptom: tsc --noEmit is clean. Ship it?

Root cause: types do not encode behavior. A function can typecheck and still drop rows. Generated casts hide the hole.

Replacement: one behavioral probe per claimed contract. Compile is lint. It is not a verdict.

# proposed probes — fill with your real domain
CONTRACTS = [
    ("refund_total", {"items": [10, 2], "tax": 0}, 12),
    ("refund_total", {"items": [], "tax": 1}, 0),
]
Enter fullscreen mode Exit fullscreen mode

If I cannot write that table, I do not have a task. I have a vibe. Would you merge a vibe from a stranger?

4. The small-diff oracle

Symptom: twelve lines changed. Feels safe.

Root cause: blast radius is not line count. One line can swap auth. One line can widen a query.

Agents love tiny lethal patches. Reviewers love tiny diffs. That pairing is how bugs ship.

Replacement: classify the diff. Fail on auth, money, delete, and schema paths. Unless a human flag is set.

# proposed pre-merge classify
git diff --name-only origin/main...HEAD \
  | rg "auth|payment|migration|schema|delete" && echo "needs-human"
Enter fullscreen mode Exit fullscreen mode

Ask yourself one question. Would I accept this from a stranger on the internet?

5. The self-reported done oracle

Symptom: the last message says "fixed." You stop.

Root cause: the model grades its own homework. Completion text is not evidence. Confident prose is still prose.

Replacement: ignore the narration. Read artifacts only. This is the only success packet I accept.

{
  "tests_collected": 14,
  "tests_failed": 0,
  "tests_skipped": 0,
  "forbidden_paths_touched": [],
  "behavior_probes_failed": 0,
  "agent_said_done": true
}
Enter fullscreen mode Exit fullscreen mode

agent_said_done is telemetry. It is never the gate. Ever. Why are you still parsing the chat?

A reproducible oracle pack

I want one command. Laptop and remote shell. Same fail.

Proposed layout:

oracle_pack/
  gate.py
  contracts.json
  frozen/
  report.json
Enter fullscreen mode Exit fullscreen mode

gate.py fails closed. Missing keys fail. Missing files fail.

#!/usr/bin/env python3
"""Proposed oracle pack. Wire it to your runner."""
from pathlib import Path
import json

FORBIDDEN = ("auth", "payment", "migration", "schema")

def load_report(path: Path) -> dict:
    if not path.exists():
        raise SystemExit("oracle-fail: missing report.json")
    return json.loads(path.read_text())

def main() -> None:
    report = load_report(Path("report.json"))
    required = (
        "tests_collected",
        "tests_failed",
        "tests_skipped",
        "touched_paths",
        "behavior_probes_failed",
    )
    for key in required:
        if key not in report:
            raise SystemExit(f"oracle-fail: missing {key}")
    if report["tests_collected"] < 1:
        raise SystemExit("oracle-fail: nothing collected")
    if report["tests_failed"] != 0:
        raise SystemExit("oracle-fail: tests failed")
    if report["tests_skipped"] == report["tests_collected"]:
        raise SystemExit("oracle-fail: all skipped")
    if report["behavior_probes_failed"] != 0:
        raise SystemExit("oracle-fail: behavior probe")
    touched = report["touched_paths"]
    hits = [p for p in touched if any(f in p.lower() for f in FORBIDDEN)]
    if hits:
        raise SystemExit(f"oracle-fail: human-required paths {hits}")
    print("oracle-ok")

if __name__ == "__main__":
    main()
Enter fullscreen mode Exit fullscreen mode

Run it like this.

python3 gate.py
echo $?
Enter fullscreen mode Exit fullscreen mode

I keep report.json as the only input. The agent may emit logs. The gate does not parse vibes.

Debug the lying oracle first

When a run "succeeds" and the product still breaks, I do not swap models. I interrogate the judge.

  1. Print collected, failed, skipped. Not the exit code.
  2. Diff testdata/frozen/ against git HEAD.
  3. Re-run probes on a clean checkout of the patch.
  4. List touched paths. Mark forbidden ones.
  5. Drop the assistant transcript from the packet.

If step two shows the agent edited goldens, I already have my bug. Why would a new endpoint fix that?

Why free endpoints make lying oracles louder

Paid models hide a bad judge with extra luck. Free endpoints usually do not.

You see more refusals. You see more partial patches. You see more "done" messages with empty diffs. That noise is useful. It is not a model verdict.

If I change the model and the same gate.py flips, then I may have a model issue. If I change the model and the gate never ran, I learned nothing. Which one were you doing last week?

Where a free coding server actually helps

I still need a cheap place to exercise the gate. Not to skip it.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode shows up here for two operator-supplied facts only. Free model access. A free server option. I do not treat those as a quality claim.

The workflow stays boring on purpose.

  1. Freeze contracts and fixtures on my machine.
  2. Point the agent at the free server.
  3. Let it patch a throwaway branch.
  4. Copy report.json back.
  5. Run gate.py locally. Same rules.

Why split the judge? Because the agent can rewrite an in-band oracle. Call that judge-in-band if you want a sixth name. Keep the gate off the model's host.

Does this prove a free model is good? No. It proves my definition of good is stable.

If you want to rehearse that split, MonkeyCode's free model access and free server are enough. That is the only invite.

Test plan I write down

No dashboard. One table. Fill it per task.

Step Command or check Pass means
Collect pytest --collect-only count > 0
Execute pytest with a JSON report failed = 0, skipped < collected
Behavior run contracts.json probes every row matches
Diff class path allow-list no forbidden path without a flag
Judge host gate.py on my laptop exit 0

If a cell is blank, I do not have a result. I have a story. Which cell is blank on your last AI PR?

Limitations

This pack will annoy you. Good.

It blocks refactors that touch auth by design. It misses logic bugs outside the probe table. It assumes you can write fixtures. It assumes a shell and pytest.

Who should not use this approach?

  • People generating sketches with no merge path
  • Teams with zero tests to freeze
  • Folks who need formal proofs, not gates
  • Anyone hoping a free endpoint replaces review

A free server does not make a weak oracle honest. It only makes an honest oracle cheap to rerun.

What I will not claim

I will not quote latency numbers. I will not quote token caps. I will not name models I did not pin.

Those figures rot by next week. These anti-patterns do not.

If your last merge was "the model said done," stop. Replace the oracle first. Then pick an endpoint.

Top comments (0)