Free Triage Bots Look Great in Demos. Count the Confusion Before You Ship.

Free model triage demos are smooth.
The model reads one issue and returns one label.
The demo shows the right label.
Everyone ships the bot.
Then the wrong calls start.

Accuracy hides those wrong calls.
I have seen real triage sets where one class is huge.
A model can label everything as 'question' and look great.
That is useful to no one.
The bug that should have been caught gets routed to docs.

I evaluate free model triage with a confusion matrix.
The method works with any OpenAI-compatible endpoint.
MonkeyCode's free model access and free server option make the test affordable.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Why accuracy lies here

Say your repo gets 80% questions.
A naive model guesses 'question' for every issue.
It scores 80% accuracy.
The demo looks impressive.
It misses every real bug.

That is not a model failure alone.
It is an evaluation failure.
You measured the wrong thing.

A confusion matrix splits every call into four buckets:

• True positive: the model says bug and it is a bug.
• False positive: the model says bug but it is a feature.
• False negative: the model says feature but it is a bug.
• True negative: the model avoids the wrong label.

For spam, false positives are worse than false negatives.
Closing a real issue as spam erodes trust in the bot.

Score each label separately

Do not report one accuracy number.
Report precision and recall per label.

• Precision answers: when the model says this label, how often is it right?
• Recall answers: how many of the real label did the model find?
• F1 is the harmonic mean of the two.

A free model may be great at spam precision and awful at bug recall.
One number will not tell you that.

A tiny harness for honest scoring

I keep two files: issues.jsonl and score_triage.py.
The first file holds 30 to 50 hand-labeled issues I already closed.
I write the labels myself.
I do not copy public issues with personal detail.

Here is a tiny set to start.

{"text": "App crashes after login", "label": "bug"}
{"text": "Add a dark mode toggle", "label": "feature"}
{"text": "How do I rotate the API key?", "label": "question"}
{"text": "Buy followers now", "label": "spam"}

The second file calls the endpoint and scores the output.

# score_triage.py
import csv
import json
import os
import sys
from collections import Counter, defaultdict

import requests

BASE_URL = os.environ['MONKEYCODE_BASE_URL'].rstrip('/')
API_KEY = os.environ['MONKEYCODE_API_KEY']
MODEL = os.environ.get('MODEL_NAME', 'monkeycode-current')
LABELS = ('bug', 'feature', 'question', 'spam')


def classify(text):
    response = requests.post(
        BASE_URL + '/chat/completions',
        headers={'Authorization': 'Bearer ' + API_KEY},
        json={
            'model': MODEL,
            'temperature': 0,
            'messages': [
                {
                    'role': 'system',
                    'content': (
                        'Classify the issue as exactly one label: '
                        'bug, feature, question, or spam. '
                        'Reply with the label only.'
                    ),
                },
                {'role': 'user', 'content': text},
            ],
        },
        timeout=20,
    )
    if response.status_code != 200:
        return 'http_' + str(response.status_code)
    content = response.json()['choices'][0]['message']['content'].strip().lower()
    for label in LABELS:
        if label in content:
            return label
    return 'unknown'


def load_labels(path):
    examples = []
    with open(path, encoding='utf-8') as source:
        for line in source:
            if line.strip():
                examples.append(json.loads(line))
    return examples


def evaluate(examples):
    confusion = defaultdict(Counter)
    for example in examples:
        predicted = classify(example['text'])
        actual = example['label']
        confusion[actual][predicted] += 1

    rows = []
    for actual in LABELS:
        tp = confusion[actual][actual]
        predicted_total = sum(confusion[a][actual] for a in LABELS)
        actual_total = sum(confusion[actual][p] for p in LABELS)
        precision = tp / predicted_total if predicted_total else 0.0
        recall = tp / actual_total if actual_total else 0.0
        f1 = 2 * precision * recall / (precision + recall) if precision + recall else 0.0
        rows.append({
            'label': actual,
            'support': actual_total,
            'precision': round(precision, 3),
            'recall': round(recall, 3),
            'f1': round(f1, 3),
        })

    writer = csv.DictWriter(
        open('triage_scores.csv', 'w', newline=''),
        fieldnames=['label', 'support', 'precision', 'recall', 'f1'],
    )
    writer.writeheader()
    writer.writerows(rows)

    print('label       support  precision  recall  f1')
    for row in rows:
        print(row['label'].ljust(10),
              str(row['support']).rjust(7),
              str(row['precision']).rjust(9),
              str(row['recall']).rjust(6),
              str(row['f1']).rjust(5))

Run it once.

python -m venv .venv && source .venv/bin/activate
pip install requests
export MONKEYCODE_BASE_URL='https://your-endpoint.example'
export MONKEYCODE_API_KEY='replace-me'
python score_triage.py issues.jsonl

The script writes triage_scores.csv.
Read it before you trust the bot.

Run it in two lanes

A hosted free endpoint is not the only variable.
The gateway can add latency or reject labels.
MonkeyCode's free server option lets me run the same harness against a self-hosted control.
Same labels, same prompt, same parser.

Start the free server, then run:

export MONKEYCODE_BASE_URL='http://localhost:8080/v1'
export MONKEYCODE_API_KEY='replace-me'
python score_triage.py issues.jsonl

Compare the two CSVs.
If the self-hosted lane beats the hosted lane on the same model name, the gateway may be the problem.
If both lanes drop together, the model behavior changed.

Read the numbers like a maintainer

Do not chase a perfect score.
Chase the failure mode you can afford to miss.

Use this decision table.

Signal	Healthy threshold	What to do when it fails
Bug recall	0.80 or higher	Read the false negatives. Narrow the prompt. Add examples.
Spam precision	0.90 or higher	If lower, people will see real issues closed as spam.
Question recall	0.85 or higher	If lower, support questions rot in the backlog.
Unknown rate	Below 2%	The reply did not contain a valid label. Tighten the system prompt.

Lower thresholds are not universal.
Adjust them for your repo.
A security fix repo cares more about bug recall than question recall.
A support-heavy repo cares more about question routing.

The confusion matrix is the artifact.
The thresholds are your policy.

Where free models break

Free model endpoints change quietly.
A label set can shift over a weekend.
Your parser accepts only four labels.
The model starts returning 'enhancement' instead of 'feature'.
Your unknown rate rises.
A weekly rerun catches this.

Rate limits can also distort your numbers.
Do not retry failures inside the scoring run.
A retry turns a timeout into a slower successful call.
That hides instability.

Do not use this harness for creative writing.
Free models are allowed to phrase summaries differently.
Exact-match logic will fail for no useful reason.

Who should skip this

Skip this if you need only one label per issue.
Skip this if you cannot hand-label 30 closed issues.
Your labels are the ground truth.
If they are vague, the scores are noise.

Skip this if you can pin an exact model version and run it entirely in-house.
Your drift surface is smaller.
But if a free model touches issue triage, count the confusion first.
A demo will not show you where the bot lies.