The Agent Wrote the Tests. They Still Echoed the Patch.

Passing tests do not prove the agent understood the contract.
They often prove the agent quoted its own patch.
That gap is the merge risk, not the model.

I keep reviewing pull requests that look fully engineered.
The prose stays calm while the checklist stays green.
So why does production still surprise the on-call engineer?

We launder chat-driven patches through review theater.
Fluency becomes evidence and echoes become fake oracles.
This catalog names four anti-patterns I keep seeing.

Each entry has symptoms, a root cause, and a replacement.
I also include a copyable oracle check and a command pin.
Treat the snippets as labeled examples, not production history.

What this is not

This is not a ranking of coding models today.
This is not a claim that agents cannot write code.
This is a review workflow you can run tomorrow.

I am not publishing timings, winners, or customer stories.
Those claims go stale and I will not invent them.
If you need signed capacity data, use your own lab.

Anti-pattern 1: Confidence as a merge gate

Symptoms

• The summary reads like a senior engineer wrote every line.
• Reviewers praise tone, structure, and seemingly clear reasoning.
• Nobody can name the invariant that actually changed here.

Root cause

Fluency is cheap while true invariants stay expensive.
A model can narrate a patch without owning its failure.
Your process then confuses narration with actual technical review.

Replacement

Require one written invariant before anyone opens the diff.
If the author cannot state it, the PR remains a spike.
Spikes get a ticket, a date, and no merge button.

INVARIANT: refunds never exceed the captured amount
PROOF: test_refund_cap in tests/billing_oracle.py
OWNER: @oncall-owner

Ask one rude question during every model-authored review.
What still fails if we delete the generated comments now?
If the answer is nothing, you reviewed a blog post.

Anti-pattern 2: Echo tests

Symptoms

• Tests arrived in the same chat as the implementation.
• Assertions repeat literals copied from the generated body.
• Renaming a constant makes the test fail before production.

Root cause

The agent tested its story, not the external contract.
The test file is a mirror sitting beside the patch.
Green then only means I agreed with myself again.

Did the test exist before the implementation existed?
If not, you probably bought an echo, not an oracle.
Oracles come from specs. Echoes come from stdout.

Replacement

Write the oracle before the implementation is allowed.
Pin expected values from the spec, never from chat output.
If you lack a spec, you still lack a real test.

Here is a labeled example, not a measured suite.

# example_oracle.py — proposal, not a production run
from decimal import Decimal

SPEC_CAPTURED = Decimal("40.00")
SPEC_REQUESTED = Decimal("99.99")
SPEC_REFUND = Decimal("40.00")  # from the billing spec, not stdout


def refund_cap(captured: Decimal, requested: Decimal) -> Decimal:
    if requested < 0:
        raise ValueError("requested must be >= 0")
    return min(captured, requested)


def test_refund_never_exceeds_capture():
    assert refund_cap(SPEC_CAPTURED, SPEC_REQUESTED) == SPEC_REFUND


def test_oracle_does_not_quote_the_function():
    source = open(__file__, encoding="utf-8").read()
    banned = "assert refund_cap(SPEC_CAPTURED, SPEC_REQUESTED) == refund_cap"
    assert banned not in source

Run the example with a boring, repeatable command.

python -m pytest example_oracle.py -q

If your only assertion is output equals output, stop.
That check is a quote. It is not a contract.

Echo versus oracle, on one page

Signal	Echo test	Oracle test
Source of expected value	Generated code or chat	Spec, RFC, or ticket
Same session as the patch	Usually yes	Must be no
Rename a literal	Test dies first	Behavior still pinned
Delete the implementation comments	Still green	Still meaningful
Who can explain a failure	The chat log	A named owner

Print that table in the PR template if you must.
I would rather see one filled row than ten adjectives.
Which column did your last AI patch actually occupy?

Anti-pattern 3: Prompt until green

Symptoms

• The chat log is longer than the resulting diff.
• Failures vanish after another polite try again.
• Nobody saved the first command that actually failed.

Root cause

You optimized the prompt instead of pinning the system.
The failure never became a hashed, repeatable command.
The next model, or the next morning, revives it.

Is the fix a new behavior, or a new pep talk?
If you cannot tell, you do not have a fix.
You have a mood swing with a green badge.

Replacement

Freeze the failing command, not the motivational thread.
Re-run that exact command after every prompt change.
If the command drifts, the supposed fix is theater.

# pin the failure; do not paraphrase it in chat
cat > /tmp/failing_cmd.sh <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
python -m pytest tests/billing_oracle.py -q
EOF
chmod +x /tmp/failing_cmd.sh
sha256sum /tmp/failing_cmd.sh
/tmp/failing_cmd.sh; echo EXIT:$?

Paste the hash into the pull request body.
Did this command change, or did the behavior change?
Only the second answer counts as an actual fix.

Keep the script in the repo when the check matters.
A gist in chat will rot before the next on-call shift.
CI should run the same bytes your laptop ran.

install -d scripts
cp /tmp/failing_cmd.sh scripts/oracle_refund.sh
git add scripts/oracle_refund.sh tests/billing_oracle.py
git commit -m "pin refund oracle and the failing command"

Anti-pattern 4: Sandbox as production proof

Symptoms

• It ran on a free box, so we called it done.
• No owner, no data class, no rollback, no page path.
• The demo path is the only path anyone executed.

Root cause

A disposable environment proves one thing: reproducibility.
It does not prove capacity, identity, or blast radius.
People collapse those proofs because the demo felt good.

Would you page this sandbox at three in the morning?
If the answer is no, it never signed production.
Stop letting a clean demo launder an unsigned change.

Replacement

Split the checks. Keep them boring and explicitly named.

Check	Sandbox may answer	Production still needs
Does the oracle fail closed?	Yes	Same oracle in CI
Can a stranger reproduce it?	Yes	README command plus hash
Will it hold real load?	No	Your own capacity test
Who gets paged at 03:00?	No	A named owner
Is this data class allowed?	No	Policy, not a chat

I use a free coding environment only as a reproduction box.
MonkeyCode is an open source option with free model access and a free server. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

That box can rerun scripts/oracle_refund.sh from a clean tree.
It cannot sign your incident, your identity, or customer data.
If you need those signatures, you already left the sandbox.

A workflow you can copy tomorrow

Do this on the next model-authored pull request.
Do not skip the writing. The writing is the review.
Empty fields mean you are still in chat-driven mode.

1. State the invariant in one plain sentence.
2. Add one oracle that never imports patch literals.
3. Pin the failing command and store its sha256.
4. Mark sandbox results as reproduced, never shipped.
5. Name an owner who can revert without the chat log.

PR template
- Invariant:
- Oracle test path:
- Failing command hash:
- Reproduced on (sandbox/CI):
- Ship check (yes/no + owner):

Fill it in public comments, not in a private sidebar.
If any field is empty, the merge is still a spike.
Call it a spike. Do not call it engineering.

Want a tiny gate before humans even look?
This hook only checks that the template fields exist.
It will not prove the oracle is wise. Nothing local can.

# scripts/check_pr_template.sh — proposal
set -euo pipefail
file="${1:-pr_body.md}"
for key in Invariant "Oracle test path" "Failing command hash" "Ship check"; do
  grep -q "^- ${key}:" "$file"
  grep -vq "^- ${key}: *$" "$file"
done

Who should not use this

Do not use this catalog as a reason to ban models.
Do not use a free server as a load or soak test.
Do not paste secrets, prod dumps, or customer records there.

Do not treat my examples as measured benchmarks either.
I did not name models, quotas, hardware, or durations.
Those numbers rot. These anti-patterns do not rot.

Skip this sermon when the change is already specified.
A owned RFC with golden tests does not need this ritual.
You already have an oracle. Keep feeding it in CI.

Also skip it for throwaway spikes you will delete today.
Just do not merge the spike because the prose sounded sure.
Sounding sure is the cheapest part of the whole stack.

Limitations

Short review questions will not catch every failure mode.
Oracles are wrong when the spec itself is wrong.
Pinned commands rot if nobody runs them inside CI.

This workflow also slows a true exploratory spike.
That cost is the point, not an accident of process.
If the slowdown hurts, you were shipping narration.

I cannot prove your billing rules from a blog example.
Your domain invariants have to come from your own spec.
Copy the method. Do not copy my refund numbers blindly.

What I actually want in review

I want a sentence a stranger can falsify later.
I want a test that would fail if the spec changed.
I want a command hash that does not depend on charm.

Did you review behavior, or did you only review tone?
Did the tests quote the patch, or pin a contract?
If you cannot tell, the agent did not fail. Review did.

Pin the command. Name the owner. Then consider merge.
That is the whole method, and it still fits in a template.
The model can draft. It still cannot sign the incident.