A GitHub Actions bot that labels, dedupes, and asks for the missing detail, plus the one instruction that finally taught it to stay quiet.
If you maintain an active GitHub repo, you already know the chore. A new issue lands with a vague title, no steps to reproduce, and three plausible meanings. You read it, add a label, ask for more detail, check whether it repeats something from last week, then do the same thing again an hour later. None of it is hard. All of it adds up, and it quietly eats the time you wanted to spend on actual code.
It is not a small problem either. In Tidelift's 2024 maintainer survey, roughly 60% of open-source maintainers said they had quit or seriously considered quitting, and the community and issue-management side of the work gets named again and again as heavier and less rewarding than writing code.
So I built a small bot to take that first pass. It reads each new issue, applies labels from a set I define, flags likely duplicates with a short reason, and asks for missing information when the description is too thin to act on. The whole thing runs inside a GitHub Actions workflow backed by a language model, so there is no server to host and nothing running between events. I built it for the FINOS git-proxy project I help maintain.
The shape: three small pieces
Underneath, it is three parts. A GitHub Actions workflow that fires when an issue is opened, a Python script that the workflow runs, and inside that script a call to a model that has been handed a set of tools it can use to act on the issue.
That last part is what makes it an agent rather than a fixed script. Instead of hard-coding "classify, then comment, then label" in a rigid order, you describe the available actions to the model as tools, and it decides which ones to call and in what order based on what the issue actually needs. A clean issue might use a single tool call. A messy one might use three. I cover the full loop and the tool definitions in the full writeup on my blog.
Teaching it to stay quiet
The first version commented too much. I had told it to always post an acknowledgment so the author knew the issue was received, and it did exactly that, including on issues that were already clear and needed no reply. Two comments on a tidy issue reads as noise, and noise trains people to ignore the bot.
The fix was not in the code at all. It was one short addition to the system prompt that gave the model permission to do nothing and made silence the preferred outcome whenever it was unsure. Models lean toward being helpful, which in practice means chatty, so telling it plainly that staying quiet is often the right answer moved its behavior more than any amount of rule-tightening did. The exact wording I used is in the post on my site, since it is short enough to copy.
One bot, Claude or GPT or Gemini
I started on a single model and one vendor's SDK, which is fine until someone wants to run the same bot on a model they already pay for. Moving the scripts onto LiteLLM let the same workflow point at Claude, GPT, or Gemini, with the provider encoded in the model string. Function calling works across all three, which is the part that mattered, since the whole bot is built on tools. The response shapes differ in a few irritating ways that I list in the full version.
The parts that bit me
A few things only turned up once it met the real world, and these are the bits I would want to read before building my own.
The bots worked on my own pushes and then failed the first time an outside contributor opened a pull request, with every secret in the environment suddenly blank. That is a deliberate GitHub security measure, not a typo in a secret name, and the fix carries a sharp safety caveat of its own. Feeding untrusted issue and PR text straight into a prompt also opens the door to prompt injection, where the text itself tries to hand the model new instructions, so a good chunk of the post is about keeping that contained.
There are smaller traps too: the Resource not accessible by integration: 403 that means your workflow token is missing a permission, and the Your credit balance is too low to access the Anthropic API message that usually is not a billing problem at all. I kept all of those debugging notes in the original post rather than padding this one, since the fixes are the useful part and they each need a paragraph.
One line I will defend
I kept the PR reviewer focused only on security, never on general code quality. Scanning a diff for hardcoded secrets or injection risks before a human looks is useful and hard to take personally. A bot grading someone's design is a good way to drive off the contributors you most want to keep, so I left that judgment to people.
If the paperwork around issues and PRs is wearing you down, this is about a weekend of setup that takes the repetitive first pass off your plate and leaves every real decision with you. The full walkthrough, with the workflow file, the tool definitions, and all the debugging, is on my blog. The code lives in agentic-repo-manager, and it is upstreamed into FINOS git-proxy in PR #1503.
If this post helped you - don't forget to star GitProxy on GitHub!
Top comments (0)