“Fixed the bug. All tests pass.”
Great.
Which tests? Did they run? Did they cover the bug—or did the agent change the test until it passed?
A co...
For further actions, you may consider blocking this person and/or reporting abuse
One thing I’d add is that even a valid test report can miss a production failure. Take a payment service using retries and idempotency keys: the unit tests might pass, but concurrent requests could still create duplicate charges if the database constraint or transaction boundary is wrong. I’d want to see the relevant integration tests against the real persistence layer, not just a green test summary. The agent needs to prove the behavior we care about, not just prove that its code passes the tests it wrote.
Exactly. A green unit test can miss the race that causes the actual problem. For that payment example, I’d want concurrent requests tested against the real database, with one intended charge and a consistent payment record as the outcome.
Right, and I’d also check the failure path, not just the successful concurrent run. For example, if the payment transaction commits but the API times out before returning the response, the client may retry while the original request has already succeeded. The idempotency record, payment state, and retry logic need to agree on the same outcome. Otherwise, you can still get duplicate side effects even when the database ends up with one payment row.
I’d also check what happens when a fix works on one service but breaks a downstream consumer. In an event-driven system, an agent might verify that a Kafka producer publishes the expected event and that the unit tests pass, but never check whether older consumers can still deserialize the new schema or handle out-of-order events. That’s the gap between verifying a change in isolation and verifying its actual impact across the system.
Good point. “The producer publishes successfully” only checks one side. The report should also say which consumers were checked and whether compatibility or event ordering remains unverified.
Exactly, and schema compatibility alone still won’t catch every downstream issue. A consumer might deserialize the event just fine but process it incorrectly because it assumes events always arrive in sequence. I’d test schema changes against the actual consumer versions and replay events with duplicated or delayed delivery. Otherwise, the producer looks healthy while the consumer quietly corrupts the business state.
The 'confirm the regression test fails before the fix' step is the one I'd harden further — an agent can also write a test that only fails against the old code by coincidence, which is why I like your 'read changes to the tests themselves' section as the real backstop. The red-flag list (skipped tests, rewritten assertions, mocks swapped for convenience) deserves its own checklist in the completion report. Do you run the fail-first verification yourself, or does the workflow enforce it automatically?
I wouldn’t call the workflow automatically enforced unless there’s a check outside the agent making that happen. I’d include the fail-first run in the task and ask for its output, then inspect whether it failed for the expected reason. Your point about coincidence matters—a red test alone isn’t enough.
That's the sharper formulation — 'enforced' needs a check outside the agent, not the agent's own report. Including the fail-first run in the task plus inspecting the failure reason closes both gaps: the run proves the test can fail, and the reason check proves it failed for the right cause. Is the reason check itself manual review, or is there a way to make that external check automatic too?
Point 3 is the one I'd underline. A regression test that never failed against the original code proves nothing.
One gap that's easy to miss between "done" and done: tests that didn't actually run. With pytest, for example, if collection fails or the selection matches nothing, you get "no tests ran" and exit code 5, and a summary that says "tests pass" can paper over that completely. So I'd add one item to your completion report: the number of tests executed, not just pass/fail. "0 passed" and "42 passed" both look green in a one-line summary.
Your list of test changes that need a reason (skips, weakened asserts, convenient mocks) also makes a good automated check: diff the tests directory separately and flag any removed assertion for human review.
Agreed—test counts belong in the report. “No tests ran” should never become “tests passed.” I also like checking the test diff separately. Flagging removed assertions gives the reviewer something specific to inspect.
Checking the test diff separately is a good addition. A removed assertion is the quietest way for a suite to go green, and it never shows up in a pass count. Putting it in front of the reviewer as its own line item makes it hard to miss. Thanks Robert!
Point 3 is the one I'd tattoo on the review process: run the new regression test against the ORIGINAL code and watch it fail. It takes two minutes and it kills the entire class of "tests that pass because they assert nothing" — the expect(result).toBeDefined() species.
The test-diff inspection in point 4 matters even more than people think. The scariest agent change I've seen wasn't a skipped test, it was an updated expectation that looked completely reasonable in the diff. Green suite, plausible commit message, and the requirement quietly renegotiated. These days I review test changes in an agent PR with the same suspicion I'd give production changes, sometimes more.
Do you make the fail-on-old-code run part of the task itself, so the agent has to report it? Or is that a step you still do by hand?
I’d make the fail-on-old-code run part of the task and require the output, while still checking why it failed. Your example of a reasonable-looking expectation change is exactly the danger: the suite stays green while the requirement quietly changes. Test edits deserve careful review too.
Section 3 asks you to confirm the regression test fails against the original implementation. That can be made checkable instead of reported: have the agent run the new test file against the pre-change commit in a separate worktree and include the failing output, so the claim arrives with a log. A second gap sits next to "which commands ran", which is where they ran. In my comparisons of agent tools, a working directory under my home folder made Claude Code load my global instruction file into every run, so results depended on a file no report mentioned. Should the completion report also name the directory and environment the checks ran in?
Yes, directory and environment belong in it too. The commit, working directory, runtime version, and loaded config can all affect the result. A separate worktree is useful evidence, provided the test fails because of the original bug rather than a setup problem.
The gap I hit was a green job after a one-line CI edit. It passed a second test path to the same pytest call, so pytest's rootdir moved up to the common parent and the service's pyproject.toml, which set asyncio_mode = auto, stopped applying. Unmarked async tests were skipped instead of run. Every check that ran passed, so a report written to your prompt would probably still have read true. The job went from 451 passed and 61 skipped to 382 passed and 146 skipped.
That’s a great example of a truthful report still giving false confidence. The jump from 61 to 146 skipped tests is the warning the green status hides. I’d add collected, passed, failed, and skipped counts, plus the config file pytest actually loaded.
The retroactive redefinition is the failure mode that ties your five points together: an agent that cannot meet the bar quietly moves the bar - weakens an assertion, swaps in a convenient mock, edits the expected value to match what it built. Two cheap guards on top of your workflow. First, write the acceptance criteria into a file the agent may read but not edit, and diff it at the end of the run; an agent that renegotiates "done" leaves a visible mark. Second, require the completion report to list commands verbatim, because the omissions are as informative as the contents - a suite you expected that is simply never mentioned is a check that was never run.
And "what could not be checked" is the only section of any agent report I trust on first read, since it is the one the agent has no incentive to fabricate.
Defining done before the agent starts is the step most teams skip, and the rest of the post depends on it. Without a written finish line, the agent's summary becomes the definition by default.
Jackson Ly's rootdir story is the clearest example in the thread. Skipped tests going from 61 to 146 with a green run is exactly the kind of evidence that looks fine in a summary and wrong in a diff. Kiell's note on pytest exit code 5 belongs in the same checklist. Launch Gate's idea of keeping acceptance criteria in a file the agent cannot modify is the one I would adopt first, because it removes the quiet option of passing a check by changing it.
On client work we treat the unverified list as part of the delivery, not an apology. If a claim has no evidence attached, it does not count toward done. Has anyone automated that unverified list as a merge blocker, as agentwares suggested?
Your comment thread has converged on four useful facts: test counts, fail-on-original, environment/config identity, and an explicit unverified list. I built a source-only experiment that binds those kinds of receipts to the requested task scope and the observed committed Git changes, while keeping semantic acceptance and actual authority as UNKNOWN: github.com/suirindo/agent-role-con...
Would you be open to a small public replay on one completed bug fix? We would use your completion-report checklist as the incumbent baseline, introduce one honest contradiction (a skipped/substituted test identity or a committed change outside the task), repair it, and publish exactly what each control caught or missed. No platform migration or superiority claim—just one reproducible comparison. A sanitized task and commit would be enough.
One addition to point 7: I treat the unverified list as a merge blocker, not just a note in the report. If a check could not run, the change is not ready to rely on, even if everything else passed. The other check I have learned to add is a fresh-environment rerun: a result the agent produced in its own working setup is not the same as the same check passing from a clean checkout or against the live endpoint. That is where I have caught configuration, credentials, and test-data differences that all the local green checks missed.