DEV Community

EGD PR Triage
EGD PR Triage

Posted on

A maintainer's guide to reviewing AI-written pull requests

Open-source maintainers have always had to separate noisy pull requests from useful ones. An AI coding assistant can now produce a tidy diff, a confident description, and even tests — and still invent an API that is not in your tree, ignore nearby conventions, or "fix" something that was never broken.

This is not an argument against AI-assisted work. Careful contributors often use an assistant as a drafting tool, then verify the output against the real codebase and stand behind the change. The job is to tell those pull requests apart from ones that were generated, pasted, and opened without that care — quickly, consistently, and without shaming anyone.

Signals: look at the submission, not the person

A real contribution usually leaves fingerprints of this project: a real issue, module, or code path in the description; a diff scoped to that purpose; style that matches nearby code; and tests that would fail if you reverted the change.

A low-effort or lightly reviewed AI-generated pull request often looks polished and still fails those checks. Look for patterns, not a single clue:

  • Boilerplate prose. The description could be pasted onto any repository ("this improves maintainability and follows best practices") and never mentions your modules.
  • Hallucinated surface area. The diff imports a package you do not use, calls a helper that is not in the tree, or edits a path that does not exist on the base branch.
  • Claims that do not match the work. The title promises a critical fix or vulnerability patch, but the issue is unrelated or missing, or the diff does not match the description.
  • Tests that do not test. Assertions on constants, mocks that bypass the code under change, or a suite that stays green if you revert the patch.
  • Scope that wandered. A "small typo fix" that reformats many files, or a burst of near-identical pull requests that share the same skeleton.
  • Skipped project process. Template questions left blank, a checklist ticked without evidence, or an obvious miss of a rule in CONTRIBUTING.

If you cannot state, in one sentence, what problem this pull request solves in this repository, treat the submission as incomplete. That describes the work, not the person who opened it.

A lightweight triage workflow

You need a first pass you can repeat on a busy day, and a way to route each pull request: review it, ask for a revision, or close it with a kind explanation.

Start with a short rubric

Score or band the submission on five dimensions:

  1. Quality. Does it build, and does it match nearby style and conventions?
  2. Understanding. Does the description show that the author looked at this codebase, or is the reasoning generic?
  3. Completeness. Is the purpose clear, is a relevant issue linked, and are docs updated when your project requires them?
  4. Tests. Are there meaningful tests for the behavior being changed?
  5. Warning signs. Are there hallucinated APIs, fabricated claims, boilerplate prose, or other low-effort markers?

Use a small scale, such as 0 to 5 on each dimension, and define what a low, mid, and high score mean in your project. Bands such as "ready for review," "needs mentoring," or "needs substantial work" are often more useful than a raw total, as long as you can explain the result.

Label the state of the work

GitHub labels can make that band visible to anyone triaging the queue. Prefer labels that describe the submission, not the author: needs-issue, needs-tests, needs-scope, needs-revision, and ready-for-review.

Avoid labels that tag a person, guess at their tools, or use shaming language on the pull request itself. "AI slop" or "spam author" does not help the next reviewer or a contributor willing to fix the work. Keep notes about suspected mass submissions in private triage.

Ask questions a template can enforce

A pull request template fills the pull request body automatically. Ask questions that are hard to answer without looking at your project:

  • What issue or discussion does this address? Please link it.
  • Which existing modules or functions did you change, and why those?
  • How did you test this? What would fail if the change were reverted?
  • If you used an AI assistant, what did you verify by hand against this repository?

Contributors who have done the work can answer these quickly. A generated description or a blank template is itself a useful signal.

Put the expectation in CONTRIBUTING and in an AI policy

GitHub surfaces a CONTRIBUTING file when someone opens a pull request, so put the rule there — or in a short policy it links to. You can welcome assistance and still require care:

AI-assisted contributions are welcome when you understand the change and can stand behind it. Please read the diff against this codebase, complete the pull request template, and include tests for the behavior you are changing. We review the submission — not the person — against these guidelines.

Adapt the wording to your project. If you require an issue before a large change, or reject wholesale reformatting without a discussion, say so. Clear rules make a later close feel fair.

Reply with specific, encouraging feedback

Assume good intent until the pattern says otherwise. Point to a concrete file, test, or convention, and offer a path to improve. "This looks generated" ends the conversation and teaches nothing. A comment like the following often gets a better second attempt:

Thanks for opening this. The description does not yet point to a specific issue or code path in this repository, and the patch calls a helper that is not in the tree. If you can link an existing issue, aim the change at the module that already handles this case, and add a test that would fail without your patch, we will be glad to take another look.

If the submission is out of scope or not a real attempt, close it promptly and explain the rule. Kind and firm can be the same comment.

Let scores advise you — and stop there

A simple score or band helps you spend review time on pull requests that are ready, and mentoring time on ones that could be. Used well, it describes the work in front of you. Used poorly, it becomes a verdict on a person. Keep three guardrails:

  • Advisory, not automatic. A low score is a reason to look more carefully or ask for a revision, not a trigger that closes the pull request on its own.
  • Submissions, never people. Score the diff, the description, and the tests. Do not score the contributor or publish implied judgments about their character or tools.
  • Your project's policy comes first. If your CONTRIBUTING file or AI policy already says how you handle assisted work, the rubric should follow that wording.

A second pair of eyes on low scores, even a brief weekly pass, catches cases a first pass might misread.

A note about PR Triage

If you want help applying a consistent first pass to a public repository, EGD PR Triage is a read-only public-PR reader that scores each pull request on a transparent 25-point rubric. Each submission is scored 0–5 on quality, authenticity, completeness, tests, and slop indicators. A human reviewer spot-checks a sample of scores, plus every low score, each week. The maintainer decides on any feedback. There is no write access, and nothing is posted to a repository without written permission.

A free 30-day pilot is currently available to US-based maintainers and projects: https://prtriage.edensgracedigital.com/#pilot.

Eden's Grace Digital team

Top comments (0)