Fresh pull requests are coming out faster than we can review them. Let's automate the boring parts using AI.
Previously, I wrote about how to build your own bot for triaging issues. Now, it's time to automate the other half of the boring, repetitive work: PR review.
If you maintain software, the first check on a new PR is roughly the same. Does the description make sense? Is it linked to an issue? Does the diff introduce anything dangerous? It's not hard, but it is repetitive, and a good fit for automation.
So I built two small GitHub Actions bots that take little job off your plate using an LLM. One checks PR description quality and points new contributors at the rules. The other scans the diff for security problems. Both run for GitProxy, a FINOS project I help maintain. This is the short version; you can check out the actual workflow files and all the code on my website.
Why use AI to review PRs
Automating this matters more now than it used to. AI writes a ton of new code, which means tons of new PRs to review. The 2025 Stack Overflow developer survey put adoption of AI coding tools at 84%, while just 33% of developers "trust" the output. More code, less trust in it, and the same number of human reviewers is a rough combination.
On the open source side, there is another alarming number. The 2024 Tidelift maintainer report found that the single biggest gap between paid and unpaid maintainers was multi-reviewer peer review: paid maintainers did it 53% of the time, unpaid ones only 27%. Most maintainers are unpaid, so the practice that is least covered across the ecosystem is exactly the one a first-pass bot can help with. The bot does not replace a human reviewer. It makes sure something looked at the PR even on the days nobody had time.
Security adds one more reason. A 2025 Veracode analysis found that close to half of AI-generated code samples contained a security flaw, and plenty of that code now arrives in PRs where a quick human review can miss the security parts entirely.
There are products that do this for you, like GitHub's own Copilot code review, CodeRabbit, and the new Agentic Workflows preview. What I built is the DIY version: a couple hundred lines you own, running in your own Actions, pointed at whichever model you already pay for.
How the two bots work
Both talk to GitHub through PyGithub and to the model through LiteLLM, which gives one OpenAI-style interface that routes to Claude, GPT, or Gemini depending on the model string. The only setup outside the repo is a single API key. They run on the same small loop: send the PR to the model, let it call a tool, perform that action, and repeat until it has nothing left to do.
.github/
workflows/
pr-quality.yml # runs the PR description bot
security-review.yml # runs the security bot
scripts/
agents/
agent.py # the shared model loop
pr_checker.py # PR description quality bot
security_review.py # diff security bot
CONTRIBUTING.md
A nice detail from the description bot: you do not need to sift through every closed PR to find first-time contributors, because GitHub already tells you via the author_association field. Reading the webhook beats building a search.
I gave the model explicit permission to do nothing, so a clean PR gets silence instead of a pointless "looks good to me." I kept the review security-only on purpose, because an opinionated bot nitpicking people's code puts off the contributors you want to keep. Meanwhile, pointing out leaked secrets before a human notices, is much harder to get upset about.
I also took prompt injection seriously: both bots wrap untrusted PR text in clear delimiters, and the tools are narrow and have limited permissions by design: the worst a successful injection can do is post a misleading comment that a human will see. The original post covers the threat model and the rest of these decisions in more depth.
The full build
The complete write-up has both workflow files, all three Python scripts, the prompt-injection defense, and the bugs I hit along the way.
If you would rather clone it than copy from a post, the standalone code lives in agentic-repo-manager. And if you want the other half of this, the companion post covers auto-triaging GitHub issues with the same setup.
Read the full write-up, with all the code, on jescalada.com.
Top comments (0)