AI-written pull requests have a particular look. The code is tidy, the names are sensible, the description is confident, and the tests are green. That polish makes them easy to approve and harder to review well.
The mistakes are still there. They just sit in different places than a rushed human change. Here's a checklist for the ten minutes you spend on an AI-written pull request, roughly in the order worth doing them.
1. Read the task before the diff
Start with what was asked: the issue, the prompt, the plan. Then ask whether the diff does that, all of it, and nothing else. Agents often solve a nearby problem, or the stated problem plus a "while I was here" refactor nobody requested.
2. Check the size against the request
A one-line bug fix shouldn't touch nine files. If the diff is much bigger than the task, ask for it to be split or trimmed before reviewing details. Large unrequested changes are where regressions hide.
3. Look at what was deleted
Reviewers read added lines. Agents also remove things: a validation check, an error branch, a log line, a test case that was "flaky." Scan the red lines on purpose.
4. Check the tests, not just the result
Green tests only mean something if they test the change. Look for:
- Tests that assert what the code does rather than what it should do
- Mocks that replace the very thing being tested
- Tests that were changed or skipped to make the suite pass
- No new test for a bug fix
5. Follow the inputs
Wherever the change reads input (request bodies, query strings, files, environment variables, model output), check that it's validated, size-limited and safely encoded on the way out. Agents tend to secure the path they were asked about and leave a parallel path open.
6. Verify every new dependency
Check that each new package exists, is the one you think it is, is maintained, and is actually needed. Models sometimes suggest package names that don't exist or are close to popular ones, which is how typosquatting gets in.
7. Look for invented APIs and options
A call to a method or a config option that sounds right but doesn't exist in your version of the library may compile in a dynamic language and fail at runtime. If you don't recognise it, check the docs.
8. Check error handling and failure paths
Look at what happens when the network call fails, the file is missing, or the value is null. Common patterns in generated code are catch-all handlers that swallow errors and fallbacks that quietly return a default instead of failing.
9. Check secrets and config
Look for hard-coded keys, tokens, URLs or credentials, test settings left on, and debug flags. Also check that nothing sensitive was added to logs.
10. Confirm what you're approving
Make sure the commit you're approving is the one that will merge, and that the tests ran on it. If the agent pushed again after your review, look again.
Make the checklist cheaper
Most of this list can be pre-checked so the human review focuses on judgement:
- Run a separate AI review of the real diff and rank findings by severity, so the reviewer starts from a list.
- Fail the change automatically when tests didn't run on the final commit.
- Flag new dependencies and deleted tests in the pull request summary.
That's how AI CodeOps works: AI agents plan, build, review and test changes on your repository, and a pull request only opens when your tests pass on the exact commit, the AI review has no open high or critical findings, and a person approves. The reviewer still makes the call; they just start with the evidence in front of them.
AI CodeOps is free during the open beta, and free for non-commercial open-source projects: https://aicodeops.dev
Top comments (0)