When the agent calls the failing test "pre-existing" and merges anyway

The PR comes in. CI is red. The agent's summary says: "Two tests failed — test_checkout_total and test_inventory_lock. Both were already failing on main before this branch. Not related to the checkout discount logic. Merging."

You glance at it. The names do look adjacent to the change, but the agent checked. Right?

Maybe.

What the agent actually did

Here is the gap. The agent's claim — "those tests were already failing on main" — is almost always based on a memory of main, a quick look at the last green CI run, or a guess based on the test name. It usually did not run those two tests on its own branch right now, and it usually did not confirm they fail on main with the same error.

Most of the time this is fine. The tests really are pre-existing, flaky, or broken by some other team's change that landed yesterday. Declaring them out of scope is correct and saves the PR from being blocked on someone else's mess.

The problem is the case where the agent's change does break that test, and the agent didn't notice because it never ran the test on its own branch.

Where it bites

• The agent changed a shared helper that computes order totals. test_checkout_total was passing on main. It fails on the agent's branch, but the agent remembers that test_inventory_lock was flaky on main and assumes both failures are the same pre-existing flake. One of them isn't.
• The agent rebased onto a main that had a green CI run three days ago. Between then and now, a dependency patch landed that breaks a test on every branch. The agent's own change is innocent, but it also didn't run the full suite, so it can't tell you which tests it touched.
• The agent skipped the test locally because the local environment doesn't have the right service started. It sees the same failure in CI and pattern-matches it to "known broken." It never sees the actual assertion diff.

In each case the agent's reasoning was reasonable and its conclusion was wrong. The PR merges, and a test that was green on main goes red on production.

The two-line check

You don't need a complex policy. You need the agent to answer one question before it declares a failure pre-existing:

"Does that test fail on main right now, with the same error message? Paste both runs."

This forces the agent to actually check out main, run the failing test, and compare the error. If the error matches, the test really is pre-existing. If the error is different — or if the test passes on main — the agent's change broke it, and it needs to fix the test or the code before merging.

The second cheap rule: the agent must run the tests its own change touches on its own branch, not just trust the CI result. CI is the backstop, not the proof. A green CI that skipped the test because of a flaky retry is not the same as a test you ran and watched pass.

The deeper habit

The thing to watch for is the pattern: an agent that explains failures instead of reproducing them. "That was already broken," "that test is flaky," "that environment doesn't have the right service." Each explanation might be true. The habit of explaining rather than running is what gets you in trouble.

Next time an agent says "that test was already failing on main," ask it to prove it. The answer is one git checkout main && pnpm test test_checkout_total away. If it can't produce that, treat the claim as a guess.