Nick Taylor

Posted on Dec 9 • Originally published at nickyt.co

Advent of AI 2025 - Day 6: Automating GitHub Issue Triage with Goose

#adventofai #goose #githubactions #ai

I've edited this post, but AI helped. These are meant to be quick posts related to the Advent of AI. I don't have time if I'm doing one of these each day to spend a couple hours on a post. 😅

The advent of AI series leverages Goose, and open source AI agent. If you've never heard of it, check it out!

block / goose

an open source, extensible AI agent that goes beyond code suggestions - install, execute, edit, and test with any LLM

goose

a local, extensible, open source AI agent that automates engineering tasks

goose is your on-machine AI agent, capable of automating complex development tasks from start to finish. More than just code suggestions, goose can build entire projects from scratch, write and execute code, debug failures, orchestrate workflows, and interact with external APIs - autonomously.

Whether you're prototyping an idea, refining existing code, or managing intricate engineering pipelines, goose adapts to your workflow and executes tasks with precision.

Designed for maximum flexibility, goose works with any LLM and supports multi-model configuration to optimize performance and cost, seamlessly integrates with MCP servers, and is available as both a desktop app as well as CLI - making it the ultimate AI assistant for developers who want to move faster and focus on innovation.

Quick Links

Need Help?

…

View on GitHub

I built an automated GitHub issue triage system using Goose for Advent of AI 2025 Day 6. The system categorizes festival feedback, assigns priorities, and welcomes first-time contributors. It runs in GitHub Actions and handles everything from label creation to posting helpful comments.

The Challenge

Day 6 of Advent of AI asked us to build a feedback management system for a fictional winter festival. The festival coordinator was drowning in feedback, ranging from "the heaters are broken" to "where's the lost and found?"

The goal was to automatically categorize issues into urgent, bug, feature, or question, and assign priority levels. I went with Goose running in GitHub Actions because it felt like the right fit for this kind of automation.

What I Built

The system automatically:

Categorizes issues (urgent, bug, feature, question)
Assigns priority levels (high, medium, low)
Detects sentiment (positive, neutral, negative)
Creates required labels if they don't exist
Posts helpful triage comments explaining the categorization
Welcomes first-time contributors with a friendly message

All of this happens within seconds of someone creating or editing an issue.

How It Works

The Workflow

The GitHub Actions workflow triggers on issue creation or edit. Here's the flow:

Installs and configures Goose CLI with OpenRouter
Checks if this is the user's first issue (no prior issues or PRs)
Posts a welcome message if it's their first contribution
Auto-creates any missing labels
Runs Goose to analyze the issue
Applies labels based on Goose's analysis
Posts a triage comment with reasoning

The Triage Prompt

Instead of hardcoding the triage logic in the workflow, I put it in a separate file at .github/triage_prompt.txt. This makes it easy to tweak the criteria without touching the workflow.

The prompt defines:

What makes something urgent vs a bug vs a feature
How to assign priority levels
When to mark sentiment
The expected JSON output format

Goose reads this template, I substitute the issue title and body, and it returns structured JSON with labels and a comment.

Bonus: First-Time Contributor Welcome

I added detection for first-time contributors. If someone has never created an issue or PR in the repo, they get a warm welcome message before the automated triage comment runs.

The detection script checks their issue count and PR count via GitHub CLI. If it's their first issue and they have no PRs, welcome message. Otherwise, skip straight to triage.

Real Examples

I created three test issues to verify everything worked:

Issue #1: Heating System Failing

Title: Heating system failing in the Main Tent
Body: The heaters stopped working tonight. Very cold, several booths reported no heat. Needs immediate attention.

Labels: urgent, priority:high, sentiment:negative

Issue #2: Photo Booth Feature Request

Title: Photo booth should print festival-themed frames  
Body: Suggestion: Add printed frames with festival graphics so people can take physical souvenirs.

Labels: feature, priority:low, sentiment:positive

Issue #3: Lost and Found Question

Title: Where is the lost & found located?
Body: Visitor found a jacket and wants to return it. Where should they go?

Labels: question, priority:low, sentiment:neutral

You can see the actual issues and workflow runs in the GitHub repo. The Actions tab shows each triage run in detail, including what Goose returned and which labels got applied.

Customization

Everything's externalized for easy editing:

Triage logic: Edit .github/triage_prompt.txt to change how issues get categorized

Welcome message: Edit .github/WELCOME_MESSAGE.md to customize the first-timer greeting

First-time detection: Edit .github/check_first_time_contributor.sh to change what counts as "first-time" (currently: first issue AND no PRs)

No YAML knowledge required to customize any of these.

Keeping It Simple

I could have gone wild with bonus features but stuck to the MVP plus one extra: the first-time contributor welcome. That felt like it added real value without overcomplicating things.

The system is production-ready. It handles errors, creates missing labels automatically, and logs everything for debugging. The only manual step is adding your OPENROUTER_API_KEY as a GitHub secret.

Win for Maintainers

As a maintainer, something like this is great. It triages and auto labels. Sure it might not get the labeling right all the time, but this removes a chunk of burden off of open source maintainers.

The Results

The system triages issues in under 2 minutes. It's running live on the repo and you can create test issues to see it in action. The documentation is thorough (maybe too thorough) and includes setup instructions, customization guides, and troubleshooting tips.

Check out the repo if you want to see the full implementation or steal the setup for your own projects.

nickytonline / advent-of-ai-2025-day-6-triager

Winter Festival Feedback Triage System 🎄

Automated GitHub issue triage system using goose to categorize and prioritize festival feedback.

Overview

This system automatically triages festival feedback issues using AI:

🏷️ Auto-categorizes: urgent, bug, feature, or question
⚡ Assigns priority: high, medium, or low
💬 Adds helpful comments: Explains the triage decision and suggests next steps
😊 Detects sentiment: positive, neutral, or negative (optional)
👋 Welcomes first-timers: Detects and welcomes first-time contributors