FAQ: The Agent Graded Its Own Homework

Have you merged tests the agent wrote for its own patch?

I hear one sentence in almost every review.

"The suite is green, so we can ship."

That sentence hides a boring, expensive bug. The agent wrote the code. Then it wrote the tests. Then it ran them on a scratch box. Who graded the homework?

The same author did.

This FAQ names five claims I keep hearing. I will show a check you can run today. No dashboards. Just git, a fingerprint, and a second test plan.

Why this keeps slipping through

Agents are fast at producing files. Your brain is slow at doubting green output. That gap is the whole problem.

A free coding session feels like a lab. It is not your CI matrix. It is not a second engineer. It is one author with a shell.

I still use a scratch box for first drafts. I do not let that box sign coverage.

Where a scratch pad actually fits

I draft throwaway patches on a free agent server.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode offers free model access and a free server option. I treat both as a notepad with a shell. Not a coverage contract. Not a release signer.

Strip every product name out. The checklist below still holds. Use any scratch environment you already have.

Myth 1: Green agent tests mean independent coverage

Claim: the tests passed, so the behavior is proven.

What I actually see: those tests were authored with the implementation. They share the same blind spots. They share the same wrong formula.

Corrected model: same-author tests are notes, not a verdict. They record what the agent believed. They do not challenge that belief.

Ask one rude question. Would this test fail if the code were wrong? If the answer is "probably not", it is a mirror. Not a net.

A tiny mirror test

# implementation.py — agent draft
def discount(price: float, pct: float) -> float:
    return price * (1 - pct)


# test_discount.py — same session
def test_discount_ten_percent():
    assert discount(100, 0.1) == 90.0

Looks clean, right? Now watch a common agent "fix".

# implementation.py — later turn, still wrong intent
def discount(price: float, pct: float) -> float:
    return price * pct


# test_discount.py — updated in the same turn
def test_discount_ten_percent():
    assert discount(100, 0.1) == 10.0

Both files moved together. The suite stayed green. The business rule died in silence. That is homework grading homework.

Myth 2: The free server is close enough to CI

Claim: it ran on a server, so the environment counts.

What I actually see: nobody pinned the OS, runtime, or lockfile. The pass happened on one sample. CI is a matrix. One sample is not a matrix.

Corrected model: a scratch box is a field note. Print a fingerprint before you trust a pass. Compare it to CI, not to memory.

#!/usr/bin/env bash
# save as scripts/env_fingerprint.sh
set -euo pipefail

python3 - <<'PY'
import hashlib, os, platform, sys, pathlib
print(f"python={sys.version.split()[0]}")
print(f"impl={platform.python_implementation()}")
print(f"os={platform.platform()}")
print(f"cwd={pathlib.Path('.').resolve()}")
for name in ("package-lock.json", "poetry.lock", "Pipfile.lock", "go.sum", "Cargo.lock"):
    p = pathlib.Path(name)
    if p.exists():
        digest = hashlib.sha256(p.read_bytes()).hexdigest()[:16]
        print(f"lock:{name}={digest}")
PY

Store that output next to the patch. If CI disagrees, the green bar lived on another planet. Do not argue with the hashes.

Myth 3: A coverage percent proves the tests think

Claim: 90% coverage means the suite is serious.

What I actually see: line coverage loves happy paths. Agents generate assertions that hug the current code. Covered lines can still be wrong lines.

Corrected model: coverage is a map of execution. It is not a map of disagreement. Independent tests try to make the code fail. Same-author tests often try to make the code look finished.

I want one mutation, not a vanity number.

# tests/test_independence_smoke.py
# Proposal: unexecuted until you wire it to your real module.
from implementation import discount


def test_discount_rejects_over_100_percent():
    # Human-written rule. The agent never saw this file.
    try:
        discount(50, 1.5)
    except ValueError:
        return
    raise AssertionError("discount accepted an impossible percent")

If that test does not exist, you do not have a second brain. You have a transcript.

Myth 4: You can skip a human test plan

Claim: the agent already listed edge cases in chat.

What I actually see: the chat list arrives after the implementation. It is a tour of what got built. It is not a contract written before the code existed.

Corrected model: write the plan first. Keep it out of the agent session. If a case was not on the plan, it is a bonus demo. Not proof.

Pre-session plan I actually paste

# test_plan.md — write this before the agent runs
Feature: order discount
Must fail when:
- percent < 0
- percent > 1
- price is negative
Must pass when:
- 0% leaves the price unchanged
- 100% yields zero
Out of scope:
- currency conversion
- tax

Then I let the agent work. Then I compare. Anything missing from test_plan.md is unpaid debt. I do not let a chat bullet retire that debt.

Myth 5: Regenerating tests is cheaper than review

Claim: if I doubt the suite, I just ask again.

What I actually see: the second generation still shares the first prompt. Same author. Same repo snapshot. Same missing rule. You paid twice for one brain.

Corrected model: regeneration is still same-author work. Review is a different role. Split the roles in git, not in vibes.

#!/usr/bin/env bash
# scripts/same_author_warn.sh
# Proposal: warn when impl and tests change together.
set -euo pipefail

changed=$(git diff --name-only origin/main...HEAD)
impl=$(echo "$changed" | grep -E '\.(py|ts|go)$' | grep -v -E '(^|/)tests?/' || true)
tests=$(echo "$changed" | grep -E '(^|/)tests?/' || true)

if [[ -n "$impl" && -n "$tests" ]]; then
  echo "WARN: implementation and tests moved in the same range."
  echo "Require a reviewer who did not write either file."
  echo "--- impl ---"
  echo "$impl"
  echo "--- tests ---"
  echo "$tests"
fi

A warning is not a blocker. It is a speed bump. I want the speed bump.

Artifact: a two-brain workflow you can copy

This is the loop I run on a scratch box. Then I repeat the checks in CI.

1. Write test_plan.md by hand. No agent in the room.
2. Start a free model session on a free server if you have one. Draft only.
3. Export the patch. Do not keep the box as source of truth.
4. Run scripts/env_fingerprint.sh. Paste the output into the PR.
5. Run agent tests once. Treat a pass as a hint.
6. Add at least one human test the agent never saw.
7. Run scripts/same_author_warn.sh against origin/main.
8. Only then start CI. CI is brain two. The agent was brain one.

Decision table I keep above my keyboard

Signal you got	What it proves	What it does not prove
Agent tests green on a free box	The draft ran somewhere	Your CI matrix, OS, or lockfile
Chat listed edge cases	The model can narrate	The cases were independently chosen
Coverage number looks high	Lines executed	Wrong lines were challenged
Agent regenerated the suite	Another same-author pass	A second reviewer existed
Human test from test_plan.md fails	The plan still has teeth	Nothing. Fix the code.

If a cell on the right is what you needed, stop. Do not ship on the left column.

Minimal CI gate

# .github/workflows/second-brain.yml
# Proposal: adapt the runners to your repo.
name: second-brain
on: pull_request
jobs:
  independent:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: fingerprint
        run: bash scripts/env_fingerprint.sh
      - name: same-author warning
        run: bash scripts/same_author_warn.sh
      - name: human tests only
        run: python -m pytest tests/human -q

Keep agent-generated tests in tests/agent if you want them. Keep human tests in tests/human. Mixing the folders mixes the brains. That is how homework grades itself again.

Limitations

This workflow does not make tests formally independent. A teammate can still rubber-stamp the human folder. A weak test_plan.md still ships weak rules.

A free server can vanish. That is fine for a draft. It is fatal if you stored the only passing run there. Fingerprints go stale when lockfiles move. Re-run them.

I am not claiming mutation testing, property tests, or a specific model quality. Those are extra tools. This article is about role split. Role split is cheaper than another prompt.

Who should not use this approach

Skip this if you already have two humans on every patch. You do not need my folders.

Do not use a scratch agent box as the only runner for regulated releases. Do not store secrets on that box. Do not treat a free session as staging.

If your product can injure someone, this FAQ is too light. Get a real test design review. Then keep the agent in the draft lane.

What I want you to ask in standup

Did the agent grade its own homework? If yes, the green bar is a diary entry. It is not a contract.

Write the plan first. Fingerprint the box. Keep one test the model never saw. Let CI be the second brain. That is the whole method.

If you already have a free agent notepad, use it for the draft. Then run this checklist on your own runners. The second brain has to live somewhere the author does not.