Green Tests Can Still Be Dead Tests: Auditing Agent-Generated Suites

A developer recently asked a coding agent to write a test suite for a function that parses ISO dates. The agent returned fourteen test cases, every one green, and the developer merged them without a second look. A week later, a colleague extended the function to accept two additional timestamp formats, and the suite still passed even though both new formats were broken. The tests had been asserting the same happy path with slightly different inputs, and none of them pinned down the edge cases the function existed to handle.

The current conversation around AI coding agents focuses on the code they generate, but the tests they write deserve the same scrutiny. A green suite from an agent is not evidence of a meaningful suite, and the distinction matters because false confidence is more dangerous than no tests at all. Mutation testing offers a rigorous way to measure the difference, and a free server with a generous token allowance makes the measurement cheap enough to run on every agent-generated suite.

Mutation testing works by introducing small, deliberate bugs into the source code and checking whether the test suite catches them. Each deliberate bug is a mutant, and the mutation score is the percentage of mutants the suite kills. A suite that scores below fifty percent is mostly decorative, no matter how many assertions it contains. The technique is well established, but it has a reputation for being slow, which is exactly where a disposable remote workspace changes the economics.

The workflow described here uses MonkeyCode, an open-source project whose current offering includes free model access and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The free model access, which includes a 10-million-token allowance at the time of writing, is enough to generate several candidate suites for the same function, and the free server provides a clean environment where mutation testing can run without touching a local machine. Both facts are current but not permanent, so verify them before building a workflow around them.

The audit follows a fixed sequence. The agent writes a test suite for a specific function, and the generated file is saved for inspection. The repository is then cloned into a fresh workspace on the free server, and a mutation testing tool runs against the agent's suite. The surviving mutants are the final output, because they show exactly which behaviors the agent's tests failed to lock down.

The script below automates the audit. It takes a repository URL, a source file, an agent-generated test file, and a mutator name, then clones the repo, installs dependencies, drops the test file into place, and runs the mutation tool.

#!/usr/bin/env bash
set -euo pipefail

# mutant_audit.sh — mutation-test a suite written by a coding agent
# usage: ./mutant_audit.sh <repo-url> <source-file> <test-file> [mutator]

REPO_URL="${1:?repo url required}"
SOURCE_FILE="${2:?source file required}"
TEST_FILE="${3:?test file required}"
MUTATOR="${4:-stryker}"
BASE="$(mktemp -d /tmp/mutant_audit.XXXXXX)"

git clone --depth 1 "$REPO_URL" "$BASE/repo" >/dev/null 2>&1
cd "$BASE/repo"

# install dependencies quietly
if [ -f package.json ]; then
  npm ci --silent
elif [ -f pyproject.toml ] || [ -f requirements.txt ]; then
  pip install -e . -q
fi

# copy the agent-generated test file into the project
if [ -d tests ]; then
  cp "$TEST_FILE" tests/
else
  cp "$TEST_FILE" .
fi

# run the configured mutation tool
case "$MUTATOR" in
  stryker) npx stryker run --mutate "$SOURCE_FILE" --reporters json ;;
  mutmut) mutmut run --paths-to-mutate "$SOURCE_FILE" ;;
  pit) mvn test org.pitest:pitest-maven:mutationCoverage ;;
esac

rm -rf "$BASE"

The typical invocation pairs an agent command with the audit script. A developer first asks the agent to write the suite, then passes the result to the script, and the surviving mutants tell the rest of the story.

your-agent "Write a thorough test suite for src/date_parser.py. Cover invalid input, leap years, timezone offsets, and empty strings." > tests/test_date_parser.py
./mutant_audit.sh https://github.com/example/repo.git src/date_parser.py tests/test_date_parser.py mutmut

The output of the mutator is a list of surviving mutants, and that list is the real deliverable. Each survivor is a behavior the agent's tests did not pin down. A common pattern is an agent that tests the happy path with different input values but never tests the failure path, so a mutant that makes the function throw on invalid input survives. Another pattern is an agent that asserts on the output format but not on the parsed value itself, so a mutant that shifts the date by one day survives. Reading the survivors is a fast education in what the agent actually understood about the function.

The mutation score also works as a feedback signal for prompt design. If the agent's suite scores below thirty percent, the prompt probably described the function too vaguely, and a more specific prompt that names the edge cases will produce a better suite. If the score is above seventy percent, the prompt style worked, and the same phrasing can be reused for the next function. This turns mutation testing into a prompt-tuning loop, which is a more rigorous version of the usual trial-and-error approach.

The approach has real limitations. Mutation testing is computationally expensive, and a large repository with a slow test suite will take hours even on a fast server. The workflow here assumes a small, self-contained function with a fast test runner, not a monolithic application with integration tests that spin up databases. Teams with large codebases should run the audit on individual modules rather than the whole repository. The script also assumes the agent can produce a test file that the project's runner can execute, which is not guaranteed for every framework or language.

A second limitation is that mutation testing measures behavioral coverage, not semantic quality. A suite can kill every mutant and still contain tests that are hard to read, tightly coupled to implementation details, or slow to run. The audit answers one question, whether the tests would notice a change in behavior, and it leaves code review for everything else. Teams that treat a high mutation score as a license to skip review will be disappointed.

The practical recommendation is to add a mutation audit to the next agent-generated test suite before merging it. The cost is a few minutes of setup and a free server run, and the payoff is knowing whether the green suite actually means anything. A developer who has watched a mutant survive an agent's suite will never trust a green checkmark the same way again.

If the audit sounds useful, MonkeyCode's free server and current token allowance are a practical place to run it, but check the project's current terms before committing a workflow to them. The green suite is the easy part; the dead tests are the part that needs a second look.