A pull request can look perfect. Clean names, tidy functions, a green CI check. And half of the new tests can still assert nothing.
AI-written code does not look sloppy. It looks like a competent person wrote it, and that is exactly the problem. Your instincts for spotting bad code are tuned to messy code, and this is not messy.
If your team merges AI-generated code, here is a checklist to run before you do. Each item also notes what a bot can check for you, so your attention goes to the things only a human can judge.
1. Does it solve the actual problem?
AI is very good at satisfying the letter of a prompt while missing the intent. Before you read the diff, read the ticket. Write down in one sentence what this change should do.
Then look at the file list. If the ticket was about one endpoint and the pull request touches nine files, ask why. Agents love to "improve" things nobody asked them to touch.
Human's job: compare the change to the intent.
Bot's job: none. This one is yours.
2. Is the diff small enough to review?
If you cannot hold the change in your head, you cannot review it. A big AI-generated diff gets a skim and a quick approval, and that is where bugs get through.
If the diff is too large, send it back and ask for smaller pieces. This is not rude. It is the same thing you would ask of a human colleague.
Human's job: push back on size.
Bot's job: none. This one is yours.
3. Do the imports and APIs actually exist?
Generated code often calls methods or options that sound right but do not exist in your installed version. Confident-looking code is not proof.
For anything unfamiliar, check it against the official docs for the version in your lockfile. Pay special attention to new dependencies the change adds. Ask yourself whether you would have chosen that package.
Human's job: verify unfamiliar calls and new packages.
Bot's job: a dependency check can flag packages that are outdated or have known vulnerabilities, so you do not have to look each one up.
4. Do the new lines have tests, and do those tests mean anything?
This is the one that gets the least attention.
Overall coverage is a project-wide number. A pull request can add a lot of untested code and barely move it, because the old, well-tested code dilutes the result. The opposite also happens: a refactor can lower overall coverage even though the new code is fine.
What you want to know is patch coverage: of the lines this pull request changed, how many are exercised by tests? That number says much more about this specific change than the project average does.
Then check what the tests actually assert. A quick trick: flip one assertion and see whether the test still passes. If it does, the test is decoration.
// Looks like a test. Proves nothing.
test("creates a user", async () => {
const user = await createUser({ name: "Sam" });
expect(user).toBeDefined();
});
Human's job: read the assertions.
Bot's job: report coverage for the changed lines separately from the project total, so a gap is visible without opening a coverage report.
5. Did it copy something that already exists?
AI does not know your codebase the way you do. It often writes a new helper next to an identical one that already lives three folders away. Every copy is a place where the next bug fix has to be made twice.
Search for similar logic before approving. A new utility function should make you ask whether one already exists.
Human's job: recognize existing helpers you know about.
Bot's job: a duplication check can flag repeated blocks the pull request introduced.
6. What happens when things go wrong?
Generated code handles the happy path beautifully. Check the unhappy paths:
- What happens with empty input or missing values?
- What happens on a timeout or a failed network call?
- Are errors swallowed silently? An empty
catchblock is a classic. - Does it log anything useful when it fails?
Human's job: think about failure cases.
Bot's job: a complexity check can point at functions that have grown many branches, which is where failure handling usually hides.
7. Did a secret slip in?
Agents copy examples. Sometimes an example contains something that looks like a real key, and sometimes it is one. Check for hardcoded tokens, passwords and connection strings in the diff.
Human's job: glance at config and environment changes.
Bot's job: a secrets scan on the changed lines is cheap and catches the obvious cases.
A checklist you can paste into your pull request template
- [ ] I can state in one sentence what this change should do
- [ ] The files changed match the task
- [ ] The diff is small enough to review properly
- [ ] New imports and API calls exist in our installed versions
- [ ] New dependencies are justified, current and safe
- [ ] Changed lines have tests that fail when the code breaks
- [ ] Nothing duplicates a helper we already have
- [ ] Empty, null and failure cases are handled
- [ ] No secrets in the diff
Let a bot do the boring half
Items 3, 4, 5, 6 and 7 are partly mechanical. A human reading the diff is a slow and tired way to check them, and the review energy is better spent on item 1 and on the design questions only you can answer.
DebtDraft is built for this. It is a GitHub Action that leaves one comment on each pull request with a health score across the signals above: complexity, test coverage, duplication, outdated dependencies and known-vulnerable dependencies. It also shows the changed lines on their own, so you can see what this specific pull request did, and a dashboard shows how your repository trends over time.
It is free for up to three repositories, and setup is one workflow step. If you try it, let us know what you think.
Your turn
What is the sneakiest problem you have seen in an AI-generated pull request? Share it with us in the comments.
Top comments (2)
The patch coverage point in section 4 is an important distinction, because a stable project-wide number can hide exactly the lines this change adds. The flip-an-assertion trick is also a practical way to check whether an assertion matters, although smoke-level tests that only verify creation does not throw need a different standard. For section 3, checking an API against the lockfile version feels more reliable than checking the latest docs, since generated code often targets a newer surface than the project actually runs. How should a large generated PR with high patch coverage but mostly toBeDefined-style assertions be scored — as untested, or should coverage and assertion quality be reported separately?
Good question! Line coverage only says a line ran, not that anything checked it, so a large PR full of toBeDefined-style assertions can show high patch coverage and still be weakly tested. Treat the two as separate signals. Patch coverage tells you which changed lines have no test touching them at all. Assertion quality needs a different check: flip the behavior (invert a condition, change a returned value) and see whether a test fails. Mutation testing automates that, and doing it by hand on the riskiest changed function is a cheap start.
On a large generated PR, I would report them separately and read patch coverage as a floor, never a pass mark. Thank you for the lockfile point on section 3 too. Checking against the installed version is the more reliable way.