Building an AI code reviewer is one thing. Trusting it with real pull requests is another.
I've been developing CodeGuard AI, a self-hosted AI-powered GitHub PR reviewer built with Java 21 and Spring Boot 3.
Instead of relying only on sample code and happy-path demonstrations, I've been testing it against deliberately created GitHub pull requests with different bugs, test scenarios, and edge cases.
What I've been testing
So far, the test lab has covered scenarios across several areas:
- Test execution and validation
Passing test suites
Genuine assertion failures
Test execution timeouts
Builds that succeed without executing tests
Unreadable test-result evidence
- Bug detection
Unsafe Optional.get() usage
Null input handling
Incorrect discount calculations
Boundary-value errors
Incorrect implementations hidden behind passing tests
- False-positive resistance
Safe Optional.orElse() usage
Correct discount calculations
Valid test assertions
Harmless code that should not trigger unnecessary findings
- Test-quality analysis
Missing regression tests
Missing edge-case coverage
Whether tests actually validate changed behavior
Some interesting findings
A few results have been particularly useful:
CodeGuard correctly identified a genuine failing assertion and showed the expected versus actual values.
It detected an incorrect discount calculation even though the existing test passed.
It correctly classified a timed-out test execution as TIMED_OUT.
It avoided flagging valid Optional.orElse() usage.
It identified missing regression coverage without claiming that the production implementation contained a confirmed bug.
However, testing also exposed areas that need improvement.
For example, some failing test scenarios received a warning instead of a blocking merge recommendation. In another scenario, test execution was unverified while the merge decision was still marked as safe.
These are exactly the kinds of inconsistencies I want to uncover before calling the product reliable.
What I'm learning
An AI code reviewer should not simply produce more findings.
It should:
Distinguish real bugs from false positives.
Understand whether tests actually executed.
Separate verified evidence from AI inference.
Avoid suggesting fixes that weaken valid tests.
Make merge recommendations consistent with verification results.
A green build does not automatically mean the tests passed. And a suggested fix is not necessarily a correct fix.
What's next?
I'm working through a 100-scenario validation plan.
The next focus areas include repository context, regression detection, team policies, suggested-patch validation, review consistency, and merge decisions.
I'm documenting the failures alongside the successful results because both matter.
CodeGuard AI is available as a source-code product for developers interested in exploring the implementation.
I'd especially like feedback from developers who maintain CI/CD pipelines or GitHub PR review workflows:
What should an AI code reviewer absolutely verify before recommending that a pull request is safe to merge?
Built with Java 21, Spring Boot 3, GitHub integration, and AI provider support.
Explore CodeGuard AI and get the source code here: https://javacoder716.gumroad.com/l/codeguard-ai?utm_source=devto&utm_medium=article&utm_campaign=codeguard_testing_100
Top comments (0)