FAQ: Five Myths About Tests the Agent Wrote for Itself

Who actually graded the agent's homework on that run?
I keep seeing the same claim in review threads.
The agent wrote tests, they passed, so we ship.

That claim is circular because the grader sat nearby.
The same session wrote the answers and the key.

Why this FAQ exists

Green test output is not independent evidence of correctness.
The same loop wrote the code and the checks.
Ask a harder question before you merge.

Did the tests constrain the implementation at all?
Or did they photograph whatever the model just invented?

I use this FAQ when the test tree appears in one burst.
New files in src/ and tests/ often share one commit.
You still need a mental model, not a screenshot.

Myth 1: Agent-written tests are a second opinion

Standup version sounds responsible and even grown-up.
The model implemented the feature, then added tests as review.

What I actually see looks much less independent.
The tests import the same brand-new module.
They assert the happy path the prompt already described.

Demote that suite before you treat it as review.
Treat those tests as documentation of intended shape.
They are not a rival reviewer with rival incentives.

Same session, same context, same hunger for green.
Would you let a candidate grade their own take-home?
I still want the tests, but I demote them hard.

They are clues about intent, not a court verdict.

Commands I run on the commit

git log --diff-filter=A --summary --since='2 hours ago'
git diff --stat HEAD~1
git log --follow -- tests/ | head

Look at which files arrived together this hour.
Implementation and tests in one commit is a smell.
That pattern is a smell, not an automatic ban.

If src and tests move in lockstep, pause there.
Ask which file could fail without the other changing.
If the answer is none, you have a mirror suite.

Myth 2: A high coverage number pins behavior

People quote coverage like it were a lock on meaning.
Coverage is high, so the agent covered the behavior.

Coverage counts executed lines, not meaningful decision branches.
A tautology still paints a source line green.
Asserting a function equals itself pins nothing real.

Coverage is a flashlight, and it is not a lock.
Ask which inputs would change the assertion today.
If you cannot name two, the test is a tour.

Proposal: scan for assertions that cannot fail

This snippet is a proposal, not a scored benchmark.
Adapt the root path before you run anything.

# proposal: tautology_scan.py
from ast import Assert, Compare, dump, parse, walk
from pathlib import Path
import sys

def is_tautology(node: Assert) -> bool:
    test = node.test
    if not isinstance(test, Compare):
        return False
    left = dump(test.left)
    rights = [dump(c) for c in test.comparators]
    return any(left == r for r in rights)

def scan(path: Path) -> list[str]:
    hits = []
    tree = parse(path.read_text(encoding="utf-8"))
    for node in walk(tree):
        if isinstance(node, Assert) and is_tautology(node):
            hits.append(f"{path}:{node.lineno}")
    return hits

if __name__ == "__main__":
    root = Path(sys.argv[1])
    for path in root.rglob("test_*.py"):
        for hit in scan(path):
            print(hit)

python3 tautology_scan.py tests

Run it against the agent's test directory only.
Empty output is not safety, only one cheap filter.
You still need failing cases that you wrote.

Myth 3: Mocks mean the agent understood the boundary

Does a MagicMock prove the adapter was actually designed?
The agent mocked the database, so the tests are hermetic.

Mocks often return the shape the implementation already assumed.
The test never met the real adapter or the real error.
Isolation can freeze a fantasy instead of a contract.

A mock is a second copy of the same story.
Compare mock payloads with a real recorded fixture.
If they only exist in the test file, be suspicious.

Hunt the stuffed animals

grep -R -n "MagicMock\|AsyncMock\|mock.patch\|monkeypatch" tests || true
grep -R -n "return_value\|side_effect" tests || true

Search for return_value lines and read them slowly.
Would that payload survive a schema change next month?
If not, the mock is a stuffed animal, not a fence.

Prefer one real fixture over five agreeable mocks.
You do not need production data for that fixture.
Store it outside the session if the loop can edit tests.

Myth 4: Hiding tests from the prompt keeps them honest

Prompt folklore says secrecy equals scientific control.
I never pasted the tests, so the model cannot cheat.

The model still writes both sides of the bargain.
It does not need your old tests to invent mirrors.
The filesystem is in the loop with the repo.

If the agent can list tests, it can match names.
Honesty is a process, not a clever prompt trick.
Hold out a check the agent cannot edit later.

I keep a tiny golden file outside the working tree.
Or I run one human test after the loop stops.

Hold-out pattern

mkdir -p /tmp/holdout
cp tests/test_invariants.py /tmp/holdout/
# run the agent against the repo only
pytest /tmp/holdout/test_invariants.py -q -ra

The hold-out file must predate the agent session.
If the agent wrote it, it is not a hold-out.
Simple question: did you type that file yesterday?

If not, it does not count as external proof.
A copied file from this afternoon still shares authorship.
Time travel in git does not create a second mind.

Myth 5: Snapshot files are regression armor

Snapshot tooling feels like a museum alarm for JSON.
The agent added snapshots, so future diffs will save us.

First snapshots can encode the first hallucination cleanly.
A golden file of wrong JSON still fails later.
That failure protects the bug, not the user.

Snapshots freeze a story, so check the story first.
Read the snapshot instead of watching only the matcher.
If you cannot explain one field, delete the file.

find . -name '*.snap' -o -name '*__snapshots__*'
find . -name '*.snap' | head | xargs -I{} head -n 40 {}

Ask three questions out loud before you commit snapshots.

• What field is required for the user contract?
• What field is incidental formatting or noise?
• What field would a wrong model still get right?

If every field is incidental, the snapshot is theater.
Theater can still fail CI and waste a whole afternoon.
That is not armor. That is costume jewelry.

Artifact: a decision table for the PR body

I keep this table next to the pull request template.
It is a workflow, not a benchmark with fake numbers.

Observation	Do not conclude	Conclude instead	Next command
Tests arrived with the impl	The suite is independent	Same author, same hour	git log --follow -- tests/
Coverage looks high	Behavior is pinned	Lines ran once	add one failing input
Mocks return nested dicts	The boundary is tested	The fantasy was serialized	grep -R return_value tests
Prompt omitted tests	Model cannot cheat	Model can still write both	hold-out file
Snapshots committed	Regressions are blocked	A story was frozen	head the snap
Collection count is zero	Nothing broke	Nothing ran	pytest --collect-only -q

Copy the table into the pull request body.
Fill the next-command column with real command output.
Output beats vibes, and that is the whole FAQ.

A full audit script you can copy

Label this as a template and adapt paths before running.

#!/usr/bin/env bash
# template: audit_agent_tests.sh
set -euo pipefail

ROOT="${1:-.}"
cd "$ROOT"

echo "== new files in the last day =="
git log --since='1 day ago' --name-only --pretty=format: | sort -u

echo "== test vs src lockstep =="
git diff --stat HEAD~1 || true

echo "== collection count =="
pytest --collect-only -q || true

echo "== skip report =="
pytest -q -rs || true

echo "== tautology scan =="
python3 tautology_scan.py tests || true

echo "== mock payload hunt =="
grep -R -n "return_value\|side_effect" tests || true

echo "== snapshot peek =="
find . -name '*.snap' -print | head

Run it twice, once on a scratch host first.
Run it again on your laptop after that.
Diff the two transcripts before you argue about coverage.

diff -u host-audit.txt laptop-audit.txt

If collection counts differ, stop the merge immediately.
If skip counts differ, stop the merge immediately.
If only the banner differs, you can keep reading.

Where a free model and free server fit

I sometimes need a throwaway box for this audit.
I do not want leftover modules from last week.
I also do not want laptop caches lying to me.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode offers free model access and a free server option.
I use that pair when I want a clean sample host.
The model proposes test files while the server runs the audit.

Then I still replay locally because the free host is one sample.
It is not CI, production, or a trophy screenshot.
I will not invent quotas, hardware, or duration claims.

Those details change, and this FAQ does not depend on them.
The method is the two transcripts and the hold-out file.
Lose the product and the checklist still stands.

Limitations

This FAQ will not catch a clever assertion.
It will not replace a human invariant test.
It will not prove a distributed system correct.

AST tautology checks miss helper functions on purpose.
They miss equality helpers that compare a value to itself.
They miss generated tests in other languages entirely.

The bash script assumes git and pytest are installed.
It assumes a Python layout many repos do not share.
Do not treat empty scanner output as a blessing.

Who should not use this approach

Do not use this checklist as a hiring score.
Do not use this checklist as a security audit.
Do not use this on repositories you cannot clone.

Do not skip CI because a scratch host was green.
If you ship firmware, write the hold-out yourself.
If you handle money, write the hold-out yourself.

If you have no tests at all, start there first.
An audit of empty tests is empty theater only.
This FAQ assumes you already have a suite to interrogate.

What I want you to remember

The agent can write tests, and that is useful.
Useful is not the same thing as independent proof.
Same author, same hour, same incentive to go green.

Hold out one check the loop cannot edit.
Replay the suite on a second machine every time.
Read the snapshots like they might be wrong today.

They might be wrong, and that is the job.
Would you merge a self-graded exam without a second booklet?
I would not, and your users would not either.

If you already have a scratch host, run the two-transcript diff tonight.