I Fed My Messiest Bug Reports to a Free AI Triage Bot. Here's the Decision Table.

Every maintainer knows the ritual. You open the queue at 8 AM, and fifteen tabs are the same bug. One has no logs. Another has a title that reads like a ransom note. The third is a feature request wearing a bug costume.

I was losing my mornings to that queue. So I ran a small experiment. Could a free AI sort the mess before I looked at it? No paid APIs. No GPU budget. Just free model access and a free server option.

This article is the runbook. The script, the thresholds, the failures, and the decision table I now use every week. If you maintain anything with an issue tracker, steal all of it.

Why Triage Is a Filter, Not a Decision

Triage is pattern matching. Duplicate titles. Missing stack traces. Feature requests disguised as bugs. Humans are great at spotting these, but the cost is attention, and attention is the scarcest resource in open source.

A triage bot does not need to be perfect. It needs to be cheap, predictable, and loud when it is wrong. Free AI is a good fit, as long as you label your confidence.

The experiment had one rule: the bot routes, humans decide. Nothing gets closed automatically.

The Four Buckets

I limited the classifier to four outputs. Fewer buckets means fewer excuses for a wrong guess.

Bucket	What it means	Example signal
needs-info	Missing logs, steps, or version	"it crashes lol"
likely-duplicate	Same symptoms as a known issue	"same as #412 but on Windows"
no-action	Question, feature request, or praise	"can you add dark mode?"
escalate	Crash, data loss, security, PII	"deleted my data"

The prompt forces the model to pick one bucket and justify it in one line.

The Script

Here is the entire loop. It reads a JSON export of issues, classifies each one, and writes a report. No framework, no dependencies beyond requests.

import json
import sys
from pathlib import Path

import requests

BUCKETS = ["needs-info", "likely-duplicate", "no-action", "escalate"]

def build_prompt(issue: dict, known_issues: list[dict]) -> str:
    known = "\n".join(
        f"#{i['number']}: {i['title']}" for i in known_issues[:5]
    )
    return f"""
You are a bug triage assistant. Classify the issue into exactly one bucket.

Buckets:
- needs-info: no logs, no reproduction steps, no version
- likely-duplicate: same symptoms as a known issue
- no-action: question, feature request, or praise
- escalate: crash, data loss, security issue, or PII

Known issues:
{known}

Issue title: {issue['title']}
Issue body: {issue['body'][:1500]}

Reply with exactly one JSON line:
{{"bucket": "one of {BUCKETS}", "confidence": 0.0, "one_line_why": "..."}}
"""

def classify(issue: dict, endpoint: str, known_issues: list[dict]) -> dict:
    payload = {
        "model": "free",
        "messages": [{"role": "user", "content": build_prompt(issue, known_issues)}],
        "temperature": 0,
    }
    response = requests.post(endpoint, json=payload, timeout=120)
    response.raise_for_status()
    text = response.json()["choices"][0]["message"]["content"]
    return json.loads(text)

def main() -> None:
    endpoint = sys.argv[1]
    export = json.loads(Path("issues.json").read_text())
    known = [i for i in export if i["state"] == "open"]
    report = []

    for issue in export:
        if issue["state"] != "open":
            continue
        try:
            result = classify(issue, endpoint, known)
            report.append({**result, "number": issue["number"]})
        except Exception as error:
            report.append({"number": issue["number"], "bucket": "human", "confidence": 0.0, "one_line_why": str(error)})

    Path("triage-report.json").write_text(json.dumps(report, indent=2))

if __name__ == "__main__":
    main()

That "model": "free" line is deliberate. MonkeyCode's free tier exposes free model access, a reported 10M token allowance, and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I am not going to tell you the quota will last forever, because quotas change. Check the current number before you schedule anything.

The free server option is where the cron job lives. One batch per night, at 3 AM, when cold starts do not matter. The queue is sorted by the time I open the laptop.

The Decision Table

Confidence changes the action. This is the part people skip, and it is the part that keeps the bot honest.

Bucket	Confidence	Action
escalate	any	Human now. Do not auto-reply.
needs-info	>= 0.8	Auto-comment with a template asking for logs
needs-info	< 0.8	Human queue
likely-duplicate	>= 0.9	Link the oldest known issue, human confirms
likely-duplicate	< 0.9	Human queue
no-action	>= 0.8	Move to discussions, human confirms at weekly review
no-action	< 0.8	Human queue
human	0.0	Human. The network failed, not the model.

No auto-close. Ever. The only action the bot takes alone is a comment template, and that is still reviewable.

What the First Run Taught Me

I ran the script against a synthetic fixture of twelve issues. Here are the first three so you can reproduce the run.

[
  {
    "number": 1,
    "title": "App crashes on startup",
    "body": "it just crashes, nothing else. please fix.",
    "state": "open"
  },
  {
    "number": 2,
    "title": "Add dark mode",
    "body": "Would be nice to have dark mode like the old app had.",
    "state": "open"
  },
  {
    "number": 3,
    "title": "Same crash as #1 on Windows 11",
    "body": "Same as #1 but I am on Windows 11. Here is the stack trace: ...",
    "state": "open"
  }
]

The model called issue 1 needs-info with 0.93 confidence. Correct. Issue 2 landed in no-action with 0.97. Also correct. Issue 3 got likely-duplicate at 0.88, which sits under my 0.9 threshold, so it went to the human queue. That is the system working as designed: it does not need to catch everything, it only needs to flag the cheap wins.

Three lessons survived the run.

1. Short bodies are the weak spot. "App crashes on startup" with no details is a coin flip between needs-info and escalate. Add the instruction that a missing body means needs-info, and the flip becomes boring. Boring is good.
2. Duplicate detection needs a seed list. With no known issues in the prompt, the model guesses. With five candidates, it links correctly most of the time.
3. Failures are not always the model's fault. One timeout produced a human row, and that is exactly what the try/except is for. A triage bot that silently eats issues is worse than no bot.

These are the same failure modes you will see, and the decision table absorbs all three.

Limitations

This is not a silver bullet. I would not run this pipeline for queues that get fewer than ten issues a week, because the setup time is longer than the time you save.

I would also not run it on security-sensitive content. You are sending issue bodies to an external API. If your project handles PII, keep the triage in-house and pay for the privilege.

Teams that need a contractual SLA should not depend on any free tier, mine included. The script is the point. The hosting is replaceable.

Finally, the bot cannot catch what the prompt does not mention. If you have a fifth bucket, add it. Keep the model at temperature: 0 for repeatable output. Crank it up and your triage gets spicy. Do not do that.

The Bottom Line

A free triage bot does not replace maintainers. It buys back the first hour of the morning. That hour is worth more than a perfect model, and you can get it for free.

The fixture and the script above are yours to copy. Run them against your own queue, keep or kill my thresholds, and see which bucket your issues actually land in.