AI coding agents are getting very good at finishing tasks.
They modify files.
Fix errors.
Write tests.
Run commands.
Then they end with something like:
All tests pass ✅
And most of us move on.
But there is an important question:
Did the agent actually run the tests it claims passed?
That sounds obvious.
It isn’t.
Because in AI-assisted development, we are starting to trust summaries instead of evidence.
“Tests Pass” Is a Claim
Imagine an agent says:
Implementation complete.
✓ Tests pass
✓ Build succeeds
✓ No lint errors
That looks reassuring.
But you still do not know:
- what command it ran
- whether it ran the full test suite
- whether some tests were skipped
- whether the command exited successfully
- whether the output came from the latest code
- whether it ran tests at all
The message is a summary.
It is not proof.
This Is a New Kind of Trust Problem
Before coding agents, developers usually ran commands directly:
npm test
You saw the output.
You saw the failures.
You saw the exit code.
With agents, the workflow can become:
Developer asks for feature
↓
Agent changes code
↓
Agent runs something
↓
Agent summarizes result
↓
Developer trusts summary
There is now another layer between you and the actual verification.
That layer can be wrong.
The Agent May Have Run Only Part of the Tests
Suppose your project has:
unit tests
integration tests
API tests
end-to-end tests
The agent runs:
npm run test:unit
Everything passes.
Then it reports:
All tests pass.
Technically, some tests passed.
But the full application was never verified.
That difference matters.
It May Be Reporting Stale Results
Another easy failure mode:
Agent runs tests
↓
Tests pass
↓
Agent changes code again
↓
Agent reports "tests pass"
The statement was true earlier.
It may no longer be true now.
Verification should happen against the final state of the code.
Passing Tests Can Still Mean the Wrong Tests
There is another problem.
AI writes the feature.
Then AI writes the tests.
Then AI runs those tests.
They pass.
Great.
Except both the implementation and the tests may share the same misunderstanding.
For example:
Wrong requirement interpretation
↓
AI writes code
↓
AI writes tests for that interpretation
↓
Tests pass
Everything is green.
The feature is still wrong.
So there are really two questions:
Did the tests run?
and
Were they the right tests?
Both matter.
Require Evidence, Not Confidence
I have started preferring a simple rule:
If an agent makes a verification claim, ask for the evidence behind it.
Instead of accepting:
Tests pass.
Ask for:
Show me:
1. The exact command you ran.
2. The exit code.
3. How many tests ran.
4. How many failed.
5. How many were skipped.
6. Whether this was run after the final code change.
Now the claim becomes inspectable.
Give Your Agent a Verification Contract
You can make this part of your normal coding-agent instructions.
For example:
Never say "tests pass" unless you actually ran the relevant test command.
When reporting verification, include:
- exact command
- exit code
- number of tests
- failures
- skipped tests
- build status
- lint/typecheck status
If you did not run something, say "not verified."
That last line is especially important:
If you did not verify it, say so.
“I did not run the integration tests” is much more useful than false confidence.
Separate Implementation From Verification
For important changes, do not let the same workflow both create and certify the result.
A stronger pattern is:
Agent implements feature
↓
Independent test command
↓
CI verifies
↓
Human reviews result
For higher-risk changes, you can go further:
Agent A → implementation
Agent B → adversarial review
CI → tests/build/security checks
Human → final approval
The important part is independence.
The system that produced the code should not be your only source of confidence that the code is correct.
CI Should Be the Source of Truth
The agent's message should be treated as useful context.
Not final authority.
For example:
Agent says:
All tests pass.
CI says:
3 integration tests failed.
Trust CI.
This is why traditional engineering systems still matter even when AI writes more of the code.
You still want:
- CI
- deterministic tests
- build checks
- type checks
- linting
- security scans
- deployment gates
AI does not replace those systems.
It makes them more important.
Be Careful With “Fixed”
The same principle applies to other agent claims.
For example:
“The bug is fixed.”
How was it verified?
“The build works.”
Which build command ran?
“There are no breaking changes.”
What compatibility checks were performed?
“The migration is safe.”
Was it tested against realistic data?
“This is secure.”
What security checks were actually run?
AI agents can sound extremely certain.
Certainty is not evidence.
A Better Completion Message
Instead of this:
Done.
Everything works and all tests pass.
I would rather see:
Implementation complete.
Verification performed:
npm test
Exit code: 0
128 tests passed
0 failed
3 skipped
npm run typecheck
Exit code: 0
npm run lint
Exit code: 0
Integration tests were NOT run.
Manual browser testing was NOT performed.
That is much more useful.
Now I know exactly what was verified.
And what was not.
Add “Not Verified” to Your Vocabulary
One thing AI-assisted development needs more of is:
Not verified.
There is nothing wrong with an agent saying:
Implemented, but not tested.
or:
Unit tests pass, integration tests not run.
or:
I could not verify this because the required service is unavailable.
Those answers are better than pretending certainty.
Good engineering is not about sounding confident.
It is about knowing what evidence you actually have.
My Simple Rule
For any important AI-generated change:
Never trust this:
“It works.”
Prefer this:
“Here is how I verified it.”
That one difference can prevent a lot of false confidence.
A Practical Checklist
Before accepting an agent's “tests pass” message, ask:
What command was executed?
You should know the exact command.
Was it the full relevant test suite?
Not just one subset.
What was the exit code?
Success should be measurable.
Were any tests skipped?
Skipped tests matter.
Was verification run after the final code change?
Not before the last edit.
Did CI confirm the result?
Prefer independent verification.
Can I inspect the output?
Evidence should be available.
Final Thought
AI coding agents are becoming very good at producing code.
But as they become more autonomous, developers need to become more careful about one thing:
verification.
The dangerous workflow is:
AI writes code
↓
AI says it works
↓
Human believes it
↓
Merge
The better workflow is:
AI writes code
↓
AI provides evidence
↓
Independent checks run
↓
Human verifies
↓
Merge
Because:
“Tests pass” is not evidence that tests passed.
It is a claim.
And in software engineering, important claims should come with receipts.
Top comments (1)
The stale-results case is the sneakiest one, because the claim was true when it was made. A cheap fix that removes most of the arguing: make the agent end with the last 20 lines of test output plus the exit code, and have a hook reject "tests pass" unless that run started after the most recent file write (compare timestamps). Then the receipts come automatically instead of being something you have to remember to ask for.
On "AI writes the feature, then the tests": one habit that helps is approving the test names before the implementation exists. The agent can fill in the bodies, but the list of behaviours comes from a person, so a shared misunderstanding has to get past a human at least once.