An AI code reviewer should do more than identify suspicious code. It should also report validation results accurately.
For this test, I added the following JUnit assertion to a GitHub pull request:
@test
void shouldDetectIncorrectExpectedValue() {
assertEquals(5, 2 + 2);
}
The test is intentionally incorrect.
What CodeGuard AI reported:
Test execution: FAILED
Tests executed: 1
Tests failed: 1
Expected: 5
Actual: 4
Suggested fix: change the expected value to 4
It also identified the failing assertion in the changed file.
One finding isn't enough to establish overall reliability, so I'm testing CodeGuard against 100 scenarios covering failure detection, test-result evidence, false positives, merge policies, and more.
The next question is just as important: should a failing test automatically block a merge?
I'm building CodeGuard AI with Java 21 and Spring Boot. The source code is available on Gumroad.
What behavior would you expect from an AI reviewer when a test suite fails?
Top comments (0)