Aditya

Posted on Jul 25

# Building an AI-Powered GitHub Issue Triage Bot with n8n, Gemini & SigNoz

#signoz #observability #ai #n8nbrightdatachallenge

🚀 Overview

Managing GitHub issues manually can become repetitive, especially when maintainers need to categorize incoming issues, determine severity, respond to contributors, and track activity.

For the SigNoz AI Agent Observability Hackathon, I built an automated GitHub Issue Triage Bot that:

Receives GitHub issue events through webhooks
Uses Gemini AI to classify issues
Automatically posts contextual responses on GitHub
Sends telemetry to SigNoz using OpenTelemetry-compatible logs
Visualizes issue trends and severity distributions through dashboards

This project was built as part of the SigNoz AI Agent Observability Hackathon. My goal was not only to automate GitHub issue triage using AI, but also to understand how observability can help monitor and debug AI-driven workflows in real-world environments.

Demo Video: https://youtu.be/4flZ2YEblZs

GitHub Repository: https://github.com/aditya-dev-prog/Dev-Rel-AI-Bot-Analytics

🏗️ Architecture

I chose n8n because it allowed me to quickly connect GitHub, Gemini, and SigNoz without building a custom orchestration layer from scratch.

Workflow Pipeline

GitHub Issue
      │
      ▼
 GitHub Webhook
      │
      ▼
   n8n Workflow
      │
      ▼
 Gemini AI Analysis
      │
      ├── GitHub Comment
      │
      └── OpenTelemetry Logs
                    │
                    ▼
                 SigNoz
                    │
                    ▼
              Dashboards

Complete n8n Workflow

This is the final workflow running in n8n that handles issue ingestion, AI classification, GitHub responses, and telemetry export.

The workflow consists of:

GitHub Webhook Trigger
JavaScript preprocessing
Gemini AI classification
Response parsing
GitHub comment generation
Telemetry export to SigNoz
Dashboard visualization

🔄 Workflow Breakdown

Step 1: Receive GitHub Issue Events

The workflow begins when GitHub sends an issue event to the webhook.

The webhook payload contains:

Issue title
Description
Issue number
Repository metadata

Step 2: AI-Powered Classification with Gemini

After receiving the GitHub issue payload, the workflow sends the issue title and description to Gemini AI for analysis.

Gemini classifies the issue into categories such as bug, feature request, or question, and also assigns a severity level.

A typical response looks like:

{
  "category": "question",
  "severity": "low",
  "suggested_reply": "Thanks for reporting this issue..."
}

During testing, I opened several sample GitHub issues to validate the workflow end-to-end. One example issue was titled "Webhook not triggering correctly". Gemini categorized it as a bug with medium severity and generated a suggested response. The workflow then automatically posted the response back to GitHub and exported the classification details to SigNoz, where I could verify the telemetry through logs and dashboards.

Step 3: Parsing the AI Response

The Gemini response is parsed and transformed into structured fields.

The workflow extracts:

Category
Severity
Suggested response

These values are later sent to both GitHub and SigNoz.

🤖 Automated GitHub Responses

After classification, the workflow posts an AI-generated response directly on the GitHub issue.

Example Responses

Examples generated by the bot:

Connectivity test confirmation
Webhook verification feedback
Operational status updates
Follow-up guidance for contributors

This provides immediate feedback to users without requiring a maintainer to respond manually.

📊 Adding Observability with SigNoz

The project was not limited to automation.

I wanted visibility into:

What categories are most common?
What severity levels occur most often?
How frequently is the bot being used?

To answer these questions, telemetry data is exported to SigNoz.

💥 What Actually Took Most of My Time

When I started this project, I expected the AI integration to be the hardest part.

It wasn't.

On the first day, I started around 9:00 AM and spent nearly 9 hours working on the project. Most of that time was not spent building the AI workflow—it was spent debugging infrastructure issues.

Some of the biggest challenges included:

n8n continuously crashing because of Docker/WSL permission issues.
Telemetry failing to reach SigNoz because the collector was not accepting data.
Invalid JSON payloads breaking OpenTelemetry log exports.
Workflow references failing after node renaming.

Ironically, building the AI workflow was easier than making the entire system observable.

Looking back, those debugging sessions taught me more than the AI integration itself.

🛠 Challenges Faced During Development

No real project is complete without debugging.

Challenge 1: Invalid Syntax While Sending Telemetry

Initially, telemetry payloads were generated using inline expressions.

This resulted in:

invalid syntax

The issue was caused by embedding n8n expressions directly inside JSON structures.

Fix

Instead of building the payload using complex inline expressions, I restructured the workflow and generated clean JSON objects before sending them to SigNoz.

Challenge 2: Referenced Node Doesn't Exist

While formatting telemetry logs, I encountered:

Referenced node doesn't exist

The workflow was attempting to access outputs from nodes that were no longer available in the execution path.

Fix

I redesigned the workflow and used a Merge node to combine data from different branches before generating telemetry.

Challenge 3: Gemini API Availability Issues

During testing, Gemini occasionally returned:

Service unavailable
This model is currently experiencing high demand

This highlighted the importance of building retry mechanisms and handling transient failures in AI-powered workflows.

Challenge 4: Docker & WSL Permission Issues

Before I could even focus on AI workflows, I spent a significant amount of time fixing local infrastructure problems.

The n8n container repeatedly entered a restart loop because of permission conflicts between Docker and the mounted WSL directories.

After investigating container logs, I discovered that n8n could not write to its configuration directory. Fixing the directory ownership resolved the issue and allowed the workflow environment to become stable.

The issue was resolved by fixing the directory ownership:

sudo chown -R 1000:1000 ~/.n8n

🔀 Final Telemetry Pipeline

After debugging, the workflow was redesigned to merge AI classification results with GitHub metadata before exporting logs.

This ensured:

Consistent telemetry
Reliable metadata
Cleaner observability pipelines

📈 Observability Results

Once telemetry reached SigNoz, logs became searchable and structured.

Structured Log Example

The logs contain:

service.name = devrel-bot-agent
issue.category = question
issue.severity = low

Additional Metadata

Telemetry also includes:

scope_name
trace_id
span_id
deployment_environment

This makes troubleshooting significantly easier.

📊 Building Dashboards

To understand issue trends, I created dashboards inside SigNoz.

Issue Category Distribution

The dashboard groups issues by:

issue.category

and visualizes category frequency.

First Dashboard Iteration

This confirmed that telemetry was being ingested correctly.

Final Dashboard

The final dashboard includes:

Issue Categories Distribution

Tracks how issues are classified by the AI.

Triage Severity Ratio

Shows severity distribution across issues.

Bot Activity Over Time

Measures automation activity and issue processing trends.

🎯 Key Learnings

This project taught me that AI automation alone is not enough.

To build production-ready AI agents, we also need:

Observability
Monitoring
Telemetry
Debugging capabilities
Reliability engineering practices

Using SigNoz alongside n8n and Gemini helped transform a simple automation workflow into an observable AI system.

🔗 Resources

Demo Video

https://youtu.be/4flZ2YEblZs

GitHub Repository

https://github.com/aditya-dev-prog/Dev-Rel-AI-Bot-Analytics

If you'd like to reproduce the project locally, the repository contains the workflow JSON and deployment configuration used during development.

🚀 Future Improvements

There are several enhancements that could make this system even more powerful:

Multi-label issue classification
Automatic issue assignment to maintainers
Slack and Discord notifications
Advanced severity scoring
OpenTelemetry traces and metrics alongside logs
Historical trend analysis dashboards

These improvements would help transform the project from an automated workflow into a more production-ready AI operations platform.

My Experience

This was my first hackathon focused on observability and OpenTelemetry, and it pushed me far beyond simply building an automation workflow.

While building the project, I learned:

How OpenTelemetry logs are structured
How to export telemetry from n8n workflows
How to visualize AI-agent activity in SigNoz
How observability helps debug automation workflows

The most challenging part was debugging telemetry payloads and fixing workflow data flow issues, but solving those problems helped me better understand production-grade AI systems.

One of the most satisfying moments during the project was seeing the first log appear inside SigNoz after hours of debugging Docker permissions, OpenTelemetry payloads, and workflow issues.

Until that point, I wasn't sure whether the problem was in n8n, the collector, networking, or the telemetry payload itself.

When the log finally appeared, it confirmed that the entire path—from GitHub → n8n → Gemini → SigNoz—was working end to end.

Conclusion

The biggest lesson from this project is that AI agents should not operate as black boxes.

Automation is useful, but observability is what makes automation trustworthy.

By combining GitHub, Gemini AI, n8n, OpenTelemetry, and SigNoz, I was able to build a workflow that not only performs automated issue triage but also provides visibility into every important decision it makes.

If you're building AI agents, don't stop at automation—make them observable.

For me, the most valuable part of this project was learning that observability is not something you add later. It becomes essential the moment an AI workflow starts making decisions automatically.

This project gave me hands-on experience with OpenTelemetry, SigNoz, AI orchestration, and debugging distributed workflows, making it one of the most educational projects I have built so far.

DEV Community