TL;DR
I inherited an open-source repo with 214 open issues, some dating back three years. Instead of declaring issue bankruptcy and mass-closing everything, I used Claude Code as a triage partner: it classified every issue, hunted duplicates, tested reproductions against the current codebase, and drafted responses I reviewed before sending. Three weeks later the backlog was down to 9 genuinely open items β and 14 of the "stale" issues turned out to be real bugs that got fixed. Here's the exact workflow, the prompts that worked, and the ones that backfired. π
The Problem
Last year I took over maintenance of a mid-sized open-source library (a CLI tool with a few thousand GitHub stars β I'll keep it unnamed so this doesn't read as a promo). The previous maintainer had burned out, and honestly, the issue tracker was the reason why.
The state of things when I got commit access:
- 214 open issues, oldest one from 3 years ago
- No labels on ~60% of them
- A mix of bug reports, feature requests, questions that were answered but never closed, and "is this project dead?" posts
- At least a dozen issues describing behavior that had already been fixed two major versions ago
The classic advice here is "issue bankruptcy": close everything, apologize, ask people to reopen. I hate that move. Every one of those issues is a person who took time to write something down. Some of them contain real bugs nobody ever read carefully. Mass-closing throws away signal along with the noise.
But triaging 214 issues manually is brutal. I estimated it honestly: reading each issue, checking whether it reproduces on the current version, searching for duplicates, writing a considered reply β that's 15β30 minutes per issue. Call it 70+ hours of the most draining work in software. That's why the backlog existed in the first place. Nobody does that work at 10pm after their day job.
So the constraint that made this interesting: could I get an AI coding agent to do the mechanical 80% of triage β classification, duplicate detection, reproduction testing β while I kept the human 20%: judgment calls and every word that actually gets posted to a real person?
How I Solved It
I used Claude Code (v2.x at the time) with the gh CLI, running locally against a full clone of the repo. The workflow had four phases.
Phase 1: Pull everything into a triage-friendly format
First, dump the entire backlog to disk so the agent can work through it without hammering the API:
gh issue list --state open --limit 300 \
--json number,title,body,labels,createdAt,comments \
> issues.json
Then I asked Claude Code to classify every issue into a strict schema. The prompt that ended up working (after a few bad iterations β more on that in Lessons) looked roughly like this:
For each issue in issues.json, output one JSON object:
- number, title
- category: bug | feature | question | support | unclear
- status_guess: likely-fixed | likely-valid | needs-repro | duplicate-of:<n> | answered
- confidence: high | medium | low
- evidence: one sentence citing the specific code, commit, or comment
that justifies status_guess. "It seems old" is not evidence.
Rules:
- If you claim likely-fixed, cite the commit or the current code path
that fixes it. Check the actual source, not just the changelog.
- If you claim duplicate-of, both issues must describe the same root
cause, not just similar symptoms.
- When in doubt, use "unclear" with low confidence. Do not guess.
That evidence field was the single highest-leverage line in the whole project. Without it, the model happily labeled things likely-fixed based on vibes. With it, every claim came with a checkable pointer: "fixed by the retry logic added in commit abc123, see src/client.ts". I could verify each one in seconds instead of re-deriving it from scratch.
The first pass took the agent a couple of hours of chugging through the codebase and git history. Results on the 214 issues:
| Category | Count |
|---|---|
| Likely fixed already | 58 |
| Answered questions, never closed | 41 |
| Duplicates | 33 |
| Needs reproduction info | 37 |
| Likely valid bugs | 26 |
| Feature requests | 19 |
Phase 2: Verify the "likely fixed" pile by actually running things
Here's where an AI coding agent beats a classification script. For the 58 "likely fixed" issues, I had Claude Code attempt to reproduce each one against the current main branch. Most of the bug reports included a command or snippet, so the loop was:
For issue #N: extract the reproduction steps from the report.
Run them against current main in a scratch directory.
Record: REPRODUCED / NOT-REPRODUCED / CANNOT-ATTEMPT (missing info).
Save the actual terminal output as proof.
This flipped the whole exercise from "the AI thinks it's fixed" to "here is the command output showing current behavior." Out of 58:
- 44 genuinely didn't reproduce β fixed somewhere along the way β
- 9 couldn't be attempted (environment-specific: Windows paths, specific locales)
- 5 still reproduced. These were issues everyone β including the previous maintainer β had assumed were ancient history. They were real, live bugs. β οΈ
The same reproduction pass on the "likely valid" pile confirmed 14 more. So the triage surfaced 19 confirmed, reproducible bugs hiding in a backlog everyone had written off as noise. That number is the reason I'll never do issue bankruptcy again.
Phase 3: Draft responses β with a hard rule
Every issue needed a human-sounding reply: "this is fixed in v4, here's the migration note", "closing as duplicate of #123", "can you share the output of X on the current version?"
I had Claude Code draft all of them in one batch, but with a rule I decided on early and never regretted:
The agent drafts. It never posts. Every single comment gets my eyes and my edit before it goes out.
Mechanically, drafts went into a directory as issue-0042.md files, and I reviewed them in batches of 20 with a checklist: is the claim true, is the tone right, does it thank the reporter. Then a tiny script posted the approved ones:
for f in approved/issue-*.md; do
n=$(basename "$f" .md | sed 's/issue-0*//')
gh issue comment "$n" --body-file "$f"
done
Reviewing and editing 200 drafts took about four evenings. Writing them from scratch would have taken weeks β and by evening three, the from-scratch versions would have degraded into copy-pasted "closing as stale π". The drafts kept the quality floor high precisely when my energy was lowest.
Why so strict about the posting rule? Because open-source triage is communication, not batch processing. An AI-generated reply that's subtly wrong about a person's bug report is worse than no reply β it tells them nobody actually read their issue. The agent got the facts onto the page; I made sure the sentences were true and kind.
Phase 4: Fix the confirmed bugs
The 19 confirmed bugs became the fun part. Each already had a verified reproduction from Phase 2, which meant each fix started from a failing test case β the ideal setup for an AI coding agent. I worked through them over two weeks, roughly one per evening: agent proposes a fix, the reproduction becomes a regression test, I review the diff.
Sixteen landed cleanly. Three needed real design decisions (behavior changes, not bugs in the strict sense), so they stayed open β but now with a clear write-up of the trade-off instead of three years of silence.
Final state after three weeks: 214 open issues β 9, all nine of which are genuine, labeled, and have a current-version reproduction or a design question attached. The repo went from "looks abandoned" to "actively maintained" without a single mass-close.
Lessons Learned
Demand evidence, not classification. The difference between a useless triage pass and a great one was one schema field:
evidence, with the rule "cite the commit or code path, or say unclear." LLMs are eager classifiers; forcing them to show their work converts confidence into verifiability.Reproduction beats opinion β and agents are great at reproduction. "The AI thinks this is fixed" is worth little. "Here's the terminal output on current main" is worth everything. If your agent can run code, make it settle claims empirically. This is the step most people skip, and it's where the 5 zombie bugs were hiding.
Your backlog is not noise. Ours was ~9% live bugs. Nineteen reproducible bugs out of 214 "stale" issues. Issue bankruptcy would have deleted all of them along with the noise. If you're about to mass-close a backlog, run a triage pass first β it's cheap now.
Keep the human at the point of contact. Letting the agent draft but never post felt slow on day one and turned out to be the whole ballgame. People can tell when a maintainer actually engaged with their issue. The goodwill from "wow, someone finally read this properly" comments was worth more than the time saved.
Batch by similarity, not by issue number. Reviewing 20 duplicate-closures in a row, then 20 fixed-in-v4 replies, is dramatically faster than context-switching per issue. I let the classification drive the review order. Obvious in hindsight; I wasted evening one doing it numerically.
What's Next
The one-time cleanup is done, but backlogs regrow. I now run the same classify-and-evidence pass over new issues once a week β ten minutes of review instead of three weeks of archaeology. The natural next step is wiring the reproduction check to trigger on new bug reports so every issue arrives pre-verified, with the same rule intact: drafts only, a human posts.
I'm also curious whether this pattern scales down: even a repo with 30 open issues probably has one or two live bugs everyone has stopped seeing.
Wrap-up
If you maintain β or just inherited β a repo with a scary issue tracker: don't declare bankruptcy yet. The tools to do respectful, evidence-based triage at scale exist now, and the mechanical part genuinely is automatable. The judgment and the kindness still have to be yours.
If this was useful, follow me here on Dev.to β I write weekly about practical AI-assisted engineering workflows: what actually works, what breaks, and the numbers behind both. And if you've cleared a monster backlog (with or without AI), I'd love to hear your approach in the comments. π¬
Top comments (0)