“Fixed the bug. All tests pass.”
Great.
Which tests? Did they run? Did they cover the bug—or did the agent change the test until it passed?
A confident completion message is easy to read as proof. But before you merge the changes, you need to understand what actually happened.
Here’s a simple workflow for checking an AI agent’s work without manually repeating every step.
1. Define “done” before it starts
“Fix the checkout bug” leaves too much open to interpretation.
Give the agent an observable outcome:
When a discount exceeds the order subtotal, the total should be zero. Add a regression test. Preserve valid discount behavior.
Now you have something concrete to verify.
For a UI task, describe the interaction. For an API task, specify the response and side effects.
Clear acceptance criteria make both implementation and verification easier.
2. Ask for evidence in the completion report
A useful report should include:
- What changed.
- Which commands ran.
- Whether they passed or failed.
- What could not be checked.
Compare these two reports:
“Everything works.”
And:
“Added a regression test for excessive discounts. The checkout test suite passed. Browser checkout was not tested because payment credentials were unavailable.”
The second report tells you what you can rely on and what still needs attention.
When your tool exposes execution logs, inspect them. A command mentioned in a summary is still a claim until you can confirm its execution.
3. Make sure the test catches the original bug
A passing test is useful only if it checks the behavior you care about.
Suppose the task is to prevent negative totals.
This assertion tells you almost nothing:
expect(result).toBeDefined();
This one checks the requirement:
expect(calculateTotal(50, 80)).toBe(0);
Where practical, confirm that the regression test fails against the original implementation and passes after the fix.
Then check nearby behavior:
expect(calculateTotal(50, 20)).toBe(30);
The goal is to demonstrate that the bug was fixed while preserving expected behavior.
4. Read changes to the tests themselves
Sometimes the implementation needs to change. Sometimes an outdated test needs to change too.
But a green result deserves closer inspection if the agent also:
- Skipped a failing test.
- Removed an important assertion.
- Replaced realistic behavior with a convenient mock.
- Changed the expected result to match the implementation.
- Disabled a check in configuration.
These changes can have valid reasons. Ask for the reason and compare it with the requirement.
“Tests pass” has limited meaning if the relevant checks were weakened.
5. Match verification to the change
Different changes need different evidence.
| Change | Useful verification |
|---|---|
| Calculation | Expected values and boundary cases |
| API endpoint | Status, response body, permissions, and stored changes |
| Database migration | Apply it to representative data and check compatibility |
| UI interaction | Run the interaction and inspect the resulting state |
| Background job | Check retries, duplicate handling, and failure behavior |
A successful build proves that the build completed. It doesn’t prove that checkout works.
A screenshot shows appearance. It doesn’t prove that the button saves anything.
Choose checks that exercise the promised behavior.
6. Look for fixes that moved the problem
A small bug can lead to a surprisingly large patch.
Before merging, ask:
- Did the agent change unrelated files?
- Did it introduce a dependency for a small task?
- Did it suppress an error instead of handling it?
- Did it remove validation to make the request succeed?
- Did it change a shared function and miss its other callers?
Start with the diff summary, then inspect the affected paths.
A large change isn’t automatically wrong. It should have an explanation proportional to its scope.
7. Treat blocked verification as unfinished verification
Sometimes the agent cannot run a check.
The database is unavailable. A dependency cannot be installed. Browser access is missing.
That is useful information.
The report should say:
“Implemented the change, but integration tests could not run because the test database was unavailable.”
Avoid turning “could not test” into “probably works.”
Record the missing check and complete it in the appropriate environment before relying on the result.
A prompt you can reuse
Add this to your next coding task:
Before editing, state the expected behavior and how you will verify it.
After editing, report:
- What changed.
- The exact checks you ran and their results.
- Whether the regression test catches the original failure, if applicable.
- Any tests or checks you changed, removed, or skipped—and why.
- What remains unverified.
Only claim a check passed if you actually ran it and observed a passing result.
This won’t eliminate mistakes. It gives you a clearer starting point for assessing the work.
Before you click merge
Ask one question:
“What evidence shows that this change meets the requirement?”
Sometimes the answer is a focused regression test. Sometimes it is an integration run, a browser interaction, or several checks together.
The important part is being able to connect the evidence to the task.
An AI agent can help you implement and verify a change. Its final message should make that verification inspectable.
What’s the biggest gap you’ve found between an agent saying “done” and the work actually being finished?
Top comments (5)
One thing I’d add is that even a valid test report can miss a production failure. Take a payment service using retries and idempotency keys: the unit tests might pass, but concurrent requests could still create duplicate charges if the database constraint or transaction boundary is wrong. I’d want to see the relevant integration tests against the real persistence layer, not just a green test summary. The agent needs to prove the behavior we care about, not just prove that its code passes the tests it wrote.
I’d also check what happens when a fix works on one service but breaks a downstream consumer. In an event-driven system, an agent might verify that a Kafka producer publishes the expected event and that the unit tests pass, but never check whether older consumers can still deserialize the new schema or handle out-of-order events. That’s the gap between verifying a change in isolation and verifying its actual impact across the system.
The 'confirm the regression test fails before the fix' step is the one I'd harden further — an agent can also write a test that only fails against the old code by coincidence, which is why I like your 'read changes to the tests themselves' section as the real backstop. The red-flag list (skipped tests, rewritten assertions, mocks swapped for convenience) deserves its own checklist in the completion report. Do you run the fail-first verification yourself, or does the workflow enforce it automatically?
The gap I hit was a green job after a one-line CI edit. It passed a second test path to the same pytest call, so pytest's rootdir moved up to the common parent and the service's pyproject.toml, which set asyncio_mode = auto, stopped applying. Unmarked async tests were skipped instead of run. Every check that ran passed, so a report written to your prompt would probably still have read true. The job went from 451 passed and 61 skipped to 382 passed and 146 skipped.
Point 3 is the one I'd underline. A regression test that never failed against the original code proves nothing.
One gap that's easy to miss between "done" and done: tests that didn't actually run. With pytest, for example, if collection fails or the selection matches nothing, you get "no tests ran" and exit code 5, and a summary that says "tests pass" can paper over that completely. So I'd add one item to your completion report: the number of tests executed, not just pass/fail. "0 passed" and "42 passed" both look green in a one-line summary.
Your list of test changes that need a reason (skips, weakened asserts, convenient mocks) also makes a good automated check: diff the tests directory separately and flag any removed assertion for human review.