DEV Community

Fazil Hasanov
Fazil Hasanov

Posted on

Building a Self-Optimizing Python AI Agent for GitHub Issue Triage

#ai #python #github #automation

GitHub issues are the lifeblood of open-source and enterprise software development. Yet, triaging them efficiently remains a persistent challenge—especially for popular repositories where issue volume can overwhelm maintainers. Manual triage is time-consuming, error-prone, and scales poorly.

Enter self-optimizing AI agents: autonomous systems that not only classify and prioritize GitHub issues but also learn from feedback and improve over time. In this article, we’ll build a Python-based AI agent that automates GitHub issue triage with continuous learning and self-optimization.

We’ll cover:

  • Setting up a GitHub API client
  • Using embeddings and vector search for semantic issue understanding
  • Implementing a self-optimizing feedback loop
  • Deploying the agent with minimal infrastructure

This isn’t just another script—it’s a production-ready system that gets smarter with every interaction.

1. Understanding the Problem: Why Traditional Triage Fails

Most GitHub issue triage tools rely on:

  • Keyword matching (e.g., "bug" → label as "bug")
  • Rule-based systems (e.g., if title contains "error" → assign to backend team)
  • Static ML models trained once and never updated

These approaches fail because:

  • Language is nuanced: "This is broken" vs. "I think there might be an issue" both imply bugs but use different phrasing.
  • Context evolves: A "performance" issue in v1.0 may mean something different in v2.0.
  • Feedback is ignored: Human corrections aren’t used to improve future predictions.

Our goal: an agent that understands intent, learns from mistakes, and adapts without retraining from scratch.

2. Architecture Overview

Here’s the high-level flow:

GitHub Issue Created → Agent Receives Webhook →
Semantic Analysis (Embeddings + Vector DB) →
Prediction (Labels, Assignees, Priority) →
Human Feedback (via GitHub Reactions/Comments) →
Model Update (Online Learning) →
Improved Future Predictions
Enter fullscreen mode Exit fullscreen mode

Key components:

  • GitHub Webhook Listener: Captures new issues in real time.
  • Embedding Engine: Converts issue text into semantic vectors.
  • Vector Database: Stores issue embeddings for fast similarity search.
  • Prediction Model: Classifies issues using embeddings and historical data.
  • Feedback Loop: Updates model weights based on human corrections.

3. Setting Up the GitHub API Client

We’ll use the PyGithub library to interact with GitHub.

Install Dependencies 

pip install PyGithub python-dotenv sentence-transformers faiss-cpu scikit-learn
Enter fullscreen mode Exit fullscreen mode

Authenticate with GitHub

Create a .env file:

GITHUB_TOKEN=your_personal_access_token
REPO_NAME=owner/repo
Enter fullscreen mode Exit fullscreen mode

Load it in Python:

from github import Github
from dotenv import load_dotenv
import os

load_dotenv()

g = Github(os.getenv("GITHUB_TOKEN"))
repo = g.get_repo(os.getenv("REPO_NAME"))
Enter fullscreen mode Exit fullscreen mode

Fetch New Issues 

def get_new_issues():
    issues = repo.get_issues(state="open", sort="created", direction="desc")
    return [issue for issue in issues if not issue.pull_request]
Enter fullscreen mode Exit fullscreen mode

Pro Tip: Use since parameter to fetch only recent issues and avoid rate limits:

from datetime import datetime, timedelta
since = datetime.now() - timedelta(days=1)
issues = repo.get_issues(state="open", since=since)

4. Semantic Understanding with Embeddings

We’ll use sentence embeddings to capture the meaning of issue titles and bodies.

Choose an Embedding Model

We’ll use all-MiniLM-L6-v2 from sentence-transformers—a lightweight, high-performance model.

from sentence_transformers import SentenceTransformer

model = SentenceTransformer('all-MiniLM-L6-v2')
Enter fullscreen mode Exit fullscreen mode

Generate Embeddings 

def get_issue_text(issue):
    return f"{issue.title}. {issue.body or ''}"

def embed_issue(issue):
    text = get_issue_text(issue)
    return model.encode(text, convert_to_tensor=False)
Enter fullscreen mode Exit fullscreen mode

Why this works: The model converts text into a 384-dimensional vector that captures semantic meaning. "Memory leak" and "high CPU usage" will have similar vectors, even if the words differ.

5. Building a Vector Database with FAISS

We’ll use FAISS (Facebook AI Similarity Search) to store and search issue embeddings efficiently.

Initialize FAISS Index 

import faiss
import numpy as np

dimension = 384  # Output dimension of our embedding model
index = faiss.IndexFlatL2(dimension)  # L2 distance for similarity
Enter fullscreen mode Exit fullscreen mode

Store Historical Issues 

def build_index():
    issues = get_new_issues()
    embeddings = np.array([embed_issue(issue) for issue in issues])
    index.add(embeddings)
    return issues  # Store issue objects for reference
Enter fullscreen mode Exit fullscreen mode

Find Similar Issues 

def find_similar_issues(new_embedding, k=5):
    distances, indices = index.search(np.array([new_embedding]), k)
    return distances[0], indices[0]
Enter fullscreen mode Exit fullscreen mode

Why FAISS?

  • Blazing fast even with millions of vectors
  • Supports GPU acceleration
  • Lightweight and embeddable

6. Making Predictions: Labels, Assignees, Priority

We’ll predict three things:

  1. Labels (e.g., "bug", "enhancement")
  2. Assignees (e.g., "backend-team")
  3. Priority (low, medium, high)

Approach: k-NN + Voting

For a new issue, we:

  1. Find the 5 most similar historical issues
  2. Aggregate their labels, assignees, and priorities
  3. Return the most frequent (mode) for each 
from collections import Counter

def predict_issue(issue):
    embedding = embed_issue(issue)
    _, indices = find_similar_issues(embedding, k=5)
    similar_issues = [historical_issues[i] for i in indices]

    # Extract labels, assignees, priorities
    labels = [label.name for issue in similar_issues for label in issue.labels]
    assignees = [issue.assignee.login for issue in similar_issues if issue.assignee]
    priorities = [get_priority(issue) for issue in similar_issues]  # Custom function

    # Predict
    pred_label = Counter(labels).most_common(1)[0][0] if labels else None
    pred_assignee = Counter(assignees).most_common(1)[0][0] if assignees else None
    pred_priority = Counter(priorities).most_common(1)[0][0] if priorities else "medium"

    return pred_label, pred_assignee, pred_priority
Enter fullscreen mode Exit fullscreen mode

Customize get_priority:

def get_priority(issue):
    if "urgent" in issue.title.lower() or "critical" in issue.title.lower():
        return "high"
    if "help" in issue.title.lower() or "question" in issue.title.lower():
        return "low"
    return "medium"

7. The Self-Optimizing Feedback Loop

This is where the agent gets smarter.

How Feedback Works

When a maintainer:

  • Adds/removes a label
  • Changes assignee
  • Adjusts priority

The agent detects the change and updates its knowledge.

Detecting Feedback via Webhooks

Use GitHub’s issues webhook event with edited action.

from flask import Flask, request

app = Flask(__name__)

@app.route('/webhook', methods=['POST'])
def webhook():
    data = request.json
    if data['action'] == 'edited':
        issue = repo.get_issue(data['issue']['number'])
        update_model(issue)
    return '', 200
Enter fullscreen mode Exit fullscreen mode

Updating the Model 


python
def update_model(issue):
    # Get current prediction
    pred_label, pred_assignee, pred_priority = predict_issue(issue)

    # Get actual (human-corrected) values
    actual_label = next((l.name for l in issue.labels if l.name in LABELS), None)
    actual_assignee = issue.assignee.login if issue.assignee else None
    actual_priority = get_priority(issue)  # Or parse from comment

    # If prediction was wrong, update index
    if actual_label and actual_label != pred_label:
        embedding = embed_issue(issue)
        index.add(np.array([embedding]))
        historical_issues.append(issue)  # Add
Enter fullscreen mode Exit fullscreen mode

Top comments (0)