DEV Community: Juan Escalada

Automating PR reviews with GitHub Actions bots

Juan Escalada — Sat, 27 Jun 2026 09:29:13 +0000

Fresh pull requests are coming out faster than we can review them. Let's automate the boring parts using AI.

Previously, I wrote about how to build your own bot for triaging issues. Now, it's time to automate the other half of the boring, repetitive work: PR review.

If you maintain software, the first check on a new PR is roughly the same. Does the description make sense? Is it linked to an issue? Does the diff introduce anything dangerous? It's not hard, but it is repetitive, and a good fit for automation.

So I built two small GitHub Actions bots that take little job off your plate using an LLM. One checks PR description quality and points new contributors at the rules. The other scans the diff for security problems. Both run for GitProxy, a FINOS project I help maintain. This is the short version; you can check out the actual workflow files and all the code on my website.

Why use AI to review PRs

Automating this matters more now than it used to. AI writes a ton of new code, which means tons of new PRs to review. The 2025 Stack Overflow developer survey put adoption of AI coding tools at 84%, while just 33% of developers "trust" the output. More code, less trust in it, and the same number of human reviewers is a rough combination.

On the open source side, there is another alarming number. The 2024 Tidelift maintainer report found that the single biggest gap between paid and unpaid maintainers was multi-reviewer peer review: paid maintainers did it 53% of the time, unpaid ones only 27%. Most maintainers are unpaid, so the practice that is least covered across the ecosystem is exactly the one a first-pass bot can help with. The bot does not replace a human reviewer. It makes sure something looked at the PR even on the days nobody had time.

Security adds one more reason. A 2025 Veracode analysis found that close to half of AI-generated code samples contained a security flaw, and plenty of that code now arrives in PRs where a quick human review can miss the security parts entirely.

There are products that do this for you, like GitHub's own Copilot code review, CodeRabbit, and the new Agentic Workflows preview. What I built is the DIY version: a couple hundred lines you own, running in your own Actions, pointed at whichever model you already pay for.

How the two bots work

Both talk to GitHub through PyGithub and to the model through LiteLLM, which gives one OpenAI-style interface that routes to Claude, GPT, or Gemini depending on the model string. The only setup outside the repo is a single API key. They run on the same small loop: send the PR to the model, let it call a tool, perform that action, and repeat until it has nothing left to do.

.github/
  workflows/
    pr-quality.yml          # runs the PR description bot
    security-review.yml     # runs the security bot
scripts/
  agents/
    agent.py                # the shared model loop
    pr_checker.py           # PR description quality bot
    security_review.py      # diff security bot
CONTRIBUTING.md

A nice detail from the description bot: you do not need to sift through every closed PR to find first-time contributors, because GitHub already tells you via the author_association field. Reading the webhook beats building a search.

I gave the model explicit permission to do nothing, so a clean PR gets silence instead of a pointless "looks good to me." I kept the review security-only on purpose, because an opinionated bot nitpicking people's code puts off the contributors you want to keep. Meanwhile, pointing out leaked secrets before a human notices, is much harder to get upset about.

I also took prompt injection seriously: both bots wrap untrusted PR text in clear delimiters, and the tools are narrow and have limited permissions by design: the worst a successful injection can do is post a misleading comment that a human will see. The original post covers the threat model and the rest of these decisions in more depth.

The full build

The complete write-up has both workflow files, all three Python scripts, the prompt-injection defense, and the bugs I hit along the way.

If you would rather clone it than copy from a post, the standalone code lives in agentic-repo-manager. And if you want the other half of this, the companion post covers auto-triaging GitHub issues with the same setup.

Read the full write-up, with all the code, on jescalada.com.

Putting an AI agent in charge of GitHub issue triage

Juan Escalada — Thu, 25 Jun 2026 08:44:32 +0000

A GitHub Actions bot that labels, dedupes, and asks for the missing detail, plus the one instruction that finally taught it to stay quiet.

If you maintain an active GitHub repo, you already know the chore. A new issue lands with a vague title, no steps to reproduce, and three plausible meanings. You read it, add a label, ask for more detail, check whether it repeats something from last week, then do the same thing again an hour later. None of it is hard. All of it adds up, and it quietly eats the time you wanted to spend on actual code.

It is not a small problem either. In Tidelift's 2024 maintainer survey, roughly 60% of open-source maintainers said they had quit or seriously considered quitting, and the community and issue-management side of the work gets named again and again as heavier and less rewarding than writing code.

So I built a small bot to take that first pass. It reads each new issue, applies labels from a set I define, flags likely duplicates with a short reason, and asks for missing information when the description is too thin to act on. The whole thing runs inside a GitHub Actions workflow backed by a language model, so there is no server to host and nothing running between events. I built it for the FINOS git-proxy project I help maintain.

The shape: three small pieces

Underneath, it is three parts. A GitHub Actions workflow that fires when an issue is opened, a Python script that the workflow runs, and inside that script a call to a model that has been handed a set of tools it can use to act on the issue.

That last part is what makes it an agent rather than a fixed script. Instead of hard-coding "classify, then comment, then label" in a rigid order, you describe the available actions to the model as tools, and it decides which ones to call and in what order based on what the issue actually needs. A clean issue might use a single tool call. A messy one might use three. I cover the full loop and the tool definitions in the full writeup on my blog.

Teaching it to stay quiet

The first version commented too much. I had told it to always post an acknowledgment so the author knew the issue was received, and it did exactly that, including on issues that were already clear and needed no reply. Two comments on a tidy issue reads as noise, and noise trains people to ignore the bot.

The fix was not in the code at all. It was one short addition to the system prompt that gave the model permission to do nothing and made silence the preferred outcome whenever it was unsure. Models lean toward being helpful, which in practice means chatty, so telling it plainly that staying quiet is often the right answer moved its behavior more than any amount of rule-tightening did. The exact wording I used is in the post on my site, since it is short enough to copy.

One bot, Claude or GPT or Gemini

I started on a single model and one vendor's SDK, which is fine until someone wants to run the same bot on a model they already pay for. Moving the scripts onto LiteLLM let the same workflow point at Claude, GPT, or Gemini, with the provider encoded in the model string. Function calling works across all three, which is the part that mattered, since the whole bot is built on tools. The response shapes differ in a few irritating ways that I list in the full version.

The parts that bit me

A few things only turned up once it met the real world, and these are the bits I would want to read before building my own.

The bots worked on my own pushes and then failed the first time an outside contributor opened a pull request, with every secret in the environment suddenly blank. That is a deliberate GitHub security measure, not a typo in a secret name, and the fix carries a sharp safety caveat of its own. Feeding untrusted issue and PR text straight into a prompt also opens the door to prompt injection, where the text itself tries to hand the model new instructions, so a good chunk of the post is about keeping that contained.

There are smaller traps too: the Resource not accessible by integration: 403 that means your workflow token is missing a permission, and the Your credit balance is too low to access the Anthropic API message that usually is not a billing problem at all. I kept all of those debugging notes in the original post rather than padding this one, since the fixes are the useful part and they each need a paragraph.

One line I will defend

I kept the PR reviewer focused only on security, never on general code quality. Scanning a diff for hardcoded secrets or injection risks before a human looks is useful and hard to take personally. A bot grading someone's design is a good way to drive off the contributors you most want to keep, so I left that judgment to people.

If the paperwork around issues and PRs is wearing you down, this is about a weekend of setup that takes the repetitive first pass off your plate and leaves every real decision with you. The full walkthrough, with the workflow file, the tool definitions, and all the debugging, is on my blog. The code lives in agentic-repo-manager, and it is upstreamed into FINOS git-proxy in PR #1503.

If this post helped you - don't forget to star GitProxy on GitHub!