AI coding assistants can write implementation code and tests in minutes. The problem is that both can look correct while the behavior that matters remains untested.
A passing test suite is useful evidence, but it isn't proof that a change is correct.
I've been thinking about this problem while building a Java-focused, AI-powered pull request reviewer. These are five questions I believe developers should ask before approving a PR, especially when AI helped write the code or its tests.
1. What behavior is this test actually verifying?
Start with the expected behavior, not the number of assertions.
Imagine a service that should reject a payment when the account has insufficient funds. A test might call the service and verify that it returns a response.
But what if the test never checks whether the payment was rejected?
The test can execute successfully without verifying the important business rule.
Ask yourself:
- What behavior should this test protect?
- What result would prove that behavior?
- Would the test fail if the behavior were broken?
If you cannot answer these questions, the test may provide less confidence than its green status suggests.
2. Could this test pass without checking anything meaningful?
Consider this JUnit example:
java
@test
void shouldRejectInvalidPayment() {
paymentService.processPayment(invalidPayment);
assertTrue(true);
}
The test passes as long as execution reaches the assertion. It doesn't establish that the payment was rejected.
A more meaningful test would verify the expected outcome, using the appropriate assertion for the application's behavior.
For example, if the contract requires an exception:
java
@test
void shouldRejectInvalidPayment() {
assertThrows(
InvalidPaymentException.class,
() -> paymentService.processPayment(invalidPayment)
);
}
This is only an illustration. The correct assertion depends on the actual application contract.
The important principle is simple: an assertion should verify the behavior the test claims to protect.
3. Are mocks hiding the behavior we need to verify?
Mocks are valuable when testing dependencies, failure conditions, and isolated units of code. But excessive mocking can make a test appear meaningful while avoiding the behavior that matters.
Suppose a test claims to verify that an order is saved correctly, but it mocks the repository and only verifies that save() was called.
That may establish that a method was invoked. It doesn't necessarily establish that the order was persisted correctly or that the resulting state is correct.
Before approving a heavily mocked test, ask:
- Is the test checking an outcome or just an interaction?
- Is the mocked dependency the behavior being tested?
- Would an incorrect implementation still satisfy the assertions?
- Should an integration test verify the real interaction between components?
Don't remove mocks indiscriminately. Choose the test boundary that matches the behavior you're trying to protect.
4. Does the test cover the change described in the PR?
A PR can change behavior without changing the test that originally covered it.
For example, a developer might change the conditions under which a user can access a resource. The existing test may still verify the successful access path while failing to check the newly introduced restriction.
Compare the change with the tests:
- Which behavior changed?
- Which existing tests exercise that behavior?
- What new failure scenario could the change introduce?
- Are the expected outcomes explicit?
This is also where repository context matters. The affected behavior might involve callers or related files that aren't part of the diff.
A reviewer should consider those relationships rather than treating each changed line as an isolated unit.
5. Would the test catch a deliberately broken implementation?
This is a useful thought experiment.
Imagine the implementation contains a defect that violates the requirement. Would the test fail?
If the answer is no, the test may not protect that requirement.
Mutation testing takes this idea further by making deliberate changes to the implementation and checking whether the tests detect them. It can provide stronger evidence than simply counting assertions, although it has its own limitations and costs.
You don't need a full mutation-testing setup to start asking the question during code review. But actually running tests against controlled mutations is a separate validation step, not something a normal passing test run proves.
The practical takeaway
Before merging a PR, especially one produced with AI assistance, ask:
- Does the test verify the intended behavior?
- Could it pass without a meaningful assertion?
- Are mocks hiding the behavior that matters?
- Do the tests cover the changed behavior and relevant regression scenarios?
- Would the tests detect a broken implementation?
These checks won't guarantee that every defect is caught. They can, however, help identify tests that create confidence without providing enough evidence.
AI can help write code and tests faster. The review process still needs to establish what those tests actually prove.
Disclosure: I'm building CodeGuard AI, a self-hosted AI-powered pull request reviewer for GitHub. It reviews PR changes alongside repository context and supports custom team review rules. It does not currently run mutation testing or guarantee detection of every weak test assertion.
Top comments (0)