AI-Generated Tests Can Make Coding Agents Worse. Here's How to Check Yours

Originally published on sergei-parfenov.com.

A bug fix can make every new test pass and still introduce a regression. Here is a deliberately constructed Python example, checked locally without an LLM.

An order filter has three requirements:

• Omit the filter, or pass None: return all orders.
• Pass an empty list: return no orders.
• Pass a list of statuses: return only matching orders.

The reported bug is that omitting the filter returns nothing. This proposed fix looks reasonable:

ORDERS = [
    {"id": 1, "status": "paid"},
    {"id": 2, "status": "pending"},
]

def filter_orders(orders, statuses=None):
    if not statuses:
        return list(orders)
    return [order for order in orders if order["status"] in statuses]

assert filter_orders(ORDERS) == ORDERS
assert filter_orders(ORDERS, ["paid"]) == [ORDERS[0]]
print("2 checks passed")

Both checks pass. Both branches of the if have been exercised. The reported symptom is fixed.

Now add the check for the second requirement:

assert filter_orders(ORDERS, []) == []

It fails. The function returns every order.

Python treats both None and [] as falsey. Our requirements give them different meanings, and the patch erases that distinction.

The connection to coding agents becomes more consequential when those checks determine what the agent does next.

On September 8, Leitian Tao and colleagues published the ExecCritic preprint. Holding the Qwen-3.5-35B-A3B Repair agent fixed, they reported these SWE-bench Verified results:

Feedback source	Tasks resolved
Initial repair, before generated-test feedback	61.2%
Tests from the base Qwen Test agent	57.3%
Tests from GPT-5.6-sol	65.3%

The weaker tests reduced the resolved rate by 3.9 percentage points. Better tests improved it.

Rates average three repair runs, reusing generated tests. Failed test qualification retains the initial patch in the all-task score. The baseline does not forbid repository tests. A separate official evaluator determines resolution. Feedback adds test-generation and revision work; compute budgets are not matched. These are the authors' results, not a benchmark replication for this article. Method and results.

A bad test can do more than miss a defect. It can give the next edit the wrong target.

Imagine adding this assertion to the filter example:

# This expectation contradicts the stated empty-list requirement.
assert filter_orders(ORDERS, []) == ORDERS

Our broken patch passes it. A correct implementation would fail it. Feed that failure into an automatic repair loop, and the loop now has a reason to damage correct behavior.

Adding assertions has strengthened the wrong interpretation.

Even the familiar “fails before the fix, passes afterward” check needs a closer look. Here is the original implementation from the fixture:

def filter_orders(orders, statuses=None):
    statuses = statuses or []
    return [order for order in orders if order["status"] in statuses]

The default-filter assertion fails against this version and passes against our proposed patch. It correctly detects the original bug. It simply cannot detect the new one.

The complete fix handles None explicitly:

def filter_orders(orders, statuses=None):
    if statuses is None:
        return list(orders)
    return [order for order in orders if order["status"] in statuses]

Running the same checks against all three implementations produces:

Implementation	Default + paid-filter checks	Those checks + empty-list check
Original	1 passes, 1 fails	2 pass, 1 fails
Plausible patch	2 pass	2 pass, 1 fails
Corrected patch	2 pass	3 pass

The runnable companion includes all three implementations, the checks, and the verified output. It uses Python’s standard library and makes no LLM or network calls.

The extra check earns its place because it distinguishes two implementations the earlier checks considered equally acceptable.

That is the question I would bring to an AI-generated test review: which plausible wrong implementation would this test reject?

For the filter, the candidate mistakes are easy to name: ignore the filter entirely, treat every missing filter as empty, or treat every empty filter as missing. They correspond to different misunderstandings of the contract. Tests that separate those cases tell us more than several additional examples of paid orders.

This is also where mutation testing can help: make small changes to the implementation and check whether the suite detects them. Inspect surviving mutations to understand what they change; some are equivalent for the supported inputs. For this fixture, changing statuses is None to not statuses is a useful manual mutation because it has a known, observable effect on required behavior.

For an agent workflow, I would make four changes.

1. Write down the expected behavior before reviewing the patch. Include the ordinary case, the reported failure, and the neighboring case most likely to be confused with it. Here, None and [] belong on separate rows. If the issue leaves that distinction unspecified, get a product decision before turning either interpretation into a test.

2. Review expected values as carefully as production code. An assertion is a claim about the product. Trace that claim to a requirement, an established compatibility promise, or an independently checked example. Copying the current output into an expected value can preserve the exact behavior you meant to question.

3. Keep an accepted regression check stable during repair. Let the agent change the implementation while a separate runner evaluates it with the reviewed tests. Protect the test command and configuration too: an unchanged test file helps little if the patch can skip its execution. If the test itself is wrong, revise and review it explicitly, then evaluate the candidate again.

4. Inspect the failure before asking the agent to fix it. An assertion showing the wrong returned orders is actionable behavioral evidence. A missing dependency, an import failure, or a command that selected zero tests needs a different response. Record what ran and why it failed.

ExecCritic separates test creation from repair, qualifies tests on the original repository, and keeps them unchanged during revision. That limits the repairer's ability to change its target. Separate contexts and permissions still cannot guarantee that both agents understood the issue correctly. Paper; released implementation.

The four steps above are a review procedure you can try in an existing project. They do not require training a model. Their value should be judged by the bugs and mistaken expectations they expose.

For your next AI-assisted fix, keep the original code, the proposed patch, and the new tests. Identify one plausible alternative implementation that violates the requirement. Run the tests against it.

If both implementations get the same green result, you have found a specific question the suite still cannot answer. Add the check that separates them, and review its expected result before trusting the next repair.

Comments

[Vinh Nguyen]

The mutation-testing note is the one part I would put a boundary on, because I ran it against your fixture. mutmut 3.7.0 on CPython 3.14.6, your corrected implementation as the only source file and just the first two checks as the suite: 5 mutants, 5 killed, 0 survivors. Same run against the plausible patch, the one with if not statuses that you already showed is wrong: also 5 mutants, 5 killed, 0 survivors. The operators it generated were is None to is not None, list(orders) to list(None), two string mutations on the status key, and in to not in — the mutation you name as the useful one, statuses is None to not statuses, is not in that set, since the operators rewrite a node in place rather than swap an identity test for a truthiness test. So the mutation score gives a perfect grade to both implementations under the suite that cannot separate them, which leaves your review question doing all the work rather than the tool.

[Sergei Parfenov]

This is a really useful catch. You’re right, mutmut can give a perfect score here while missing the semantic mutation that actually matters. That makes the “plausible wrong implementation” check more important than the mutation score itself. Would you treat mutation tools as a supplement, with semantic mutants derived from the contract?

[Yevhen Tienkaiev]

The distinction between a test that verifies the patch’s intent and one that verifies the contract is the key failure mode. I’d add one small review artifact for agent loops: keep the requirement, the assertion, and the observed output as separate records.

Before an agent writes or repairs a test, state the invariants and at least one neighboring case that could be confused with the reported bug. Then review the expected value against the requirement—not the current implementation—and run one deliberate negative or mutation case. If the test never goes red against a plausible wrong implementation, it has not demonstrated that it distinguishes the contract.

The same boundary shows up in browser agents: “the click returned successfully” is only an interaction result. The workflow should separately record the intended target, the page/context actually used, the action evidence, and an authoritative read-back of the postcondition. If navigation, reconnect or account drift makes that evidence stale, the result should be unknown, not a green success or an automatic retry.

Disclosure: I maintain Hronaut, a source-available local visible-browser workspace with local MCP and human takeover for sign-in, 2FA, CAPTCHA and consequential writes. I’m sharing it as one example of this evidence boundary, not as a claim that it solves the broader testing problem.

[Sergei Parfenov]

I like the idea of keeping the requirement, assertion, and observed output separate. It makes the evidence boundary much clearer and stops the current implementation from quietly becoming the oracle. Have you found a lightweight way to enforce this in agent loops without adding too much review overhead?

[howcani howcani]

Your closing question also has a countable answer, and the count decides whether the next assertion is worth reading.

I ran your fixture against eight wrong implementations, one per contract axis instead of one per token: return orders instead of list(orders); not statuses; filter ignored entirely; order["status"] == statuses; reversed output; one pass per requested status (so ["paid", "paid"] returns duplicates); a re-sort of the caller's list; and append-a-sentinel (every returned value is correct, the caller's list silently grows). Rejected:

suite	wrong implementations rejected
your two checks	3/8
+ the empty-list check	4/8
+ f(ORDERS) is not ORDERS	5/8
+ order preservation	7/8
+ "the caller's list is unchanged"	8/8

Two things fall out. The value of an added assertion is exactly the number of enumerated wrong implementations that only it rejects - my duplicate-status check (f(ORDERS, ["paid", "paid"]) == [ORDERS[0]]) rejected nothing the order check did not already reject, so on this fixture it is decoration. And the surviving axes are the ones operators do not generate: list(orders) to return orders is a one-click diff that passes your two checks, your third, and the fails-before/passes-after test; only an identity check, or a caller that mutates what it got back, separates them. Your fix writes list(orders) deliberately and nothing in the fixture observes that decision.

That is also the reading I would take from vinhnguyenthanhdn's mutmut run above: 5 mutants, 5 killed, identical for your corrected implementation and for the if not statuses version you already showed is wrong. Token-level operators on a five-line function mostly produce mutants that raise (list(None)) or move a string literal; they do not produce "right symptom, wrong contract", which is the failure mode this post is about. A 100% mutation score here is not evidence about the suite, it is evidence that the mutants were drawn from the code rather than from the requirements.

For the agent loop I would upgrade step 1 accordingly: write the expected behavior down as three to five separable wrong implementations per requirement, with the axis named - empty vs. missing, identity, order, side effects on inputs - and treat the suite as acceptable only if it rejects each of them. The list is cheap to keep next to the issue, and it tells you which assertion to add: the one that rejects an entry no current check rejects.

[Sergei Parfenov]

This is a much stronger framing than my original step 1. I like the idea of treating each requirement as a small set of named wrong implementations, then adding assertions only when they kill a survivor. That also explains why 100% mutation coverage can still say almost nothing about semantic coverage here.

Would you keep those wrong implementations as explicit fixtures in the repo, or generate them from the contract during the agent loop?

[Mike Dabydeen]

The None versus empty list case is a good fixture precisely because the defect is not in the code. It is in a requirement that was never written down anywhere the patch author could see it. Two falsy values carrying different meanings tends to live in someone's head until an incident drags it out.

Your review question, which plausible wrong implementation would this test reject, is close to a working definition of what a test is for, and it is the part I will carry into teaching. I mark a lot of student test suites, and the most common failure has never been too few assertions. It is assertions derived from the code that was just written rather than from the behaviour that was asked for. All of them pass. None of them can fail. Agents did not invent that habit, they just made it fast enough to fill a repository with it before anyone reads a line.

Of your four steps I would weight the third heaviest. Once the repairer can edit both the implementation and the check, the loop is no longer converging on the requirement, it is converging on internal consistency, and internal consistency is cheap. Protecting the test command and the configuration alongside the file is the detail most setups miss, because skipping execution looks identical to passing in the log.

One question on the ExecCritic numbers. Do you read the drop from the weaker test agent as bad tests actively misdirecting repair, or as the loop spending part of its budget on test generation instead of repair attempts? With compute unmatched those are hard to separate, and they point at different fixes.

[Sergei Parfenov]

I don’t think the numbers let us separate those two effects. The loop clearly got worse with weaker test feedback, but with unmatched compute we can’t attribute the full drop to active misdirection. I’d read it as evidence that generated-test feedback can hurt in this setup, not proof of the mechanism. A matched-budget ablation would be the interesting next experiment.

[Raknaos]

I run a fleet of autonomous agents and this matches what I've seen concretely. The failure isn't that the tests are wrong - it's that a test which verifies the patch's intent (as the LLM wrote it) is indistinguishable from one that verifies the patch's contract. Your example is the classic case: if not statuses returns orders when the spec says omit-the-filter should too, so the generated test happily locks in the bug because it asserts the flawed behavior.

What I'd add: in agent loops I've started checking the test itself against the spec, not against the code. Ask the agent to state the two invariants before writing the assertion, then diff the assertion against those invariants. Also, meta-reviewing a minimal failing test (RED first) catches a lot of these - if the agent never writes a RED test it's usually pure confirmation.

[Sergei Parfenov]

Exactly. I like the RED-first idea because it forces the test to prove it can distinguish something before the repair starts. Have you automated the invariant-to-assertion check, or is it still a review step in your loop?