An LLM Labeler Is a Function: Test It Like One

A free AI endpoint is a function. It takes text in, returns text out. Yet most of us treat it like a magic box and deploy after five manual tests.

That breaks. Model outputs shift between releases. Prompt phrasing alters behavior. A labeler that works today can quietly mislabel tomorrow. This article shows how to build a small regression harness before you trust a free-tier LLM to touch real issues, PRs, or comments.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The problem: silent output drift

A classification prompt returns "bug" one week and "question" the next, for the same input. You won't notice until a user complains. Or a mislabeled issue sits in the wrong board for days.

The fix is not better prompts. The fix is measurement. You need a fixed test set, a reproducible runner, and a threshold that tells you when the endpoint changed.

This harness uses MonkeyCode's free model endpoint for the classification calls. You can run it manually from your laptop, or schedule it on MonkeyCode's free server option so drift checks run every night without a bill.

The harness, in three pieces

A regression suite has three components:

1. A labeled dataset: 50 short texts with expected categories.
2. A runner: Python code that calls the model, collects outputs, and compares them.
3. A report: accuracy, per-category errors, and a pass/fail verdict.

No ML framework needed. Just Python, requests, and a CSV file.

The dataset: 45 realistic inputs

Size matters less than coverage. I use 45 examples because it forces me to include edge cases without drowning the free tier. Categories match a typical support triage system:

• bug: "Clicking save throws TypeError"
• feature: "Add dark mode support"
• question: "Can I use this with Django?"
• docs: "The README doesn't say how to install"

Store them as CSV. Include a mix of short titles and longer bodies. Add ambiguous cases like "does not work" (a bug or a question?

The whole set stays under 5,000 tokens. That matters when you have a generous but finite free quota.

The runner code

Here is the complete harness. It reads a CSV, calls the endpoint, and prints a confusion table.

import csv
import json
import os
import sys

import requests

API_URL = os.environ["MONKEYCODE_API_URL"]
MODEL_NAME = os.environ.get("MODEL_NAME", "")  # optional
LABELS = ["bug", "feature", "question", "docs"]


def classify(text: str) -> str:
    prompt = (
        "Classify this support message. Reply with one word. "
        f"Choose only from: {', '.join(LABELS)}.\n\n{text[:800]}"
    )
    payload = {"prompt": prompt, "max_tokens": 5}
    if MODEL_NAME:
        payload["model"] = MODEL_NAME

    resp = requests.post(API_URL, json=payload, timeout=30)
    resp.raise_for_status()
    answer = resp.json()["choices"][0]["text"].strip().lower()

    for label in LABELS:
        if label in answer:
            return label
    return "unknown"


def load_expected(path: str) -> list[dict]:
    rows = []
    with open(path, newline="") as f:
        reader = csv.DictReader(f)
        for row in reader:
            rows.append({"text": row["text"], "expected": row["expected"]})
    return rows


def run(path: str) -> dict:
    cases = load_expected(path)
    matrix = {expected: {actual: 0 for actual in LABELS + ["unknown"]}
              for expected in LABELS}
    errors = []

    for i, case in enumerate(cases, 1):
        actual = classify(case["text"])
        matrix[case["expected"]][actual] += 1
        if actual != case["expected"]:
            errors.append((i, case["expected"], actual, case["text"][:60]))

    correct = sum(matrix[k][k] for k in LABELS)
    total = len(cases)
    accuracy = correct / total if total else 0

    return {"accuracy": accuracy, "matrix": matrix, "errors": errors}


if __name__ == "__main__":
    report = run(os.environ.get("DATASET", "cases.csv"))
    print(f"Accuracy: {report['accuracy']:.0%}")
    print("\nConfusion matrix (expected -> actual):")
    for expected in LABELS:
        line = " ".join(f"{actual}:{report['matrix'][expected][actual]}"
                        for actual in LABELS + ["unknown"])
        print(f"{expected:10} | {line}")
    for index, expected, actual, snippet in report["errors"]:
        print(f"\nCase {index}: expected {expected}, got {actual}")
        print(f"  Context: {snippet}...")

Save the script as eval_harness.py. Create cases.csv with columns text,expected. Run it once to establish a baseline.

Scoring rules and a pull-the-plug threshold

Accuracy alone is not enough. Different errors have different costs. Mislabeling a bug as a feature hides a blocker. Mislabeling a question as docs is annoying but cheap.

Use this decision table to make the verdict meaningful:

Metric	Acceptance threshold	Why
Overall accuracy	≥ 95%	Below that, the endpoint is not usable for auto-triage.
bug recall	≥ 90%	Missing a bug has the highest support cost.
unknown rate	≤ 2%	Too many unknowns means the prompt is confusing the model.
Baseline stability	no regression > 5 points	Compare against the last run, not just the label.
P95 latency	< 10 s	Free endpoints can slow down; your webhook should not time out.

If a run fails the table, do not deploy or keep running the bot. Review the prompt, adjust the examples, and re-run. This is the equivalent of a unit test gate, but for a model.

Sample report output

Here is what the report material looks like for a healthy run. Your numbers will differ.

Accuracy: 98%

Confusion matrix (expected -> actual):
bug        | bug:11 feature:0 question:1 docs:0 unknown:0
feature    | bug:0 feature:9 question:1 docs:0 unknown:0
question   | bug:0 feature:0 question:12 docs:1 unknown:0
docs       | bug:0 feature:0 question:0 docs:10 unknown:0

Case 17: expected bug, got question
  Context: The form breaks when I submit an empty value...

One error out of 45 is acceptable at the 95% threshold. Two errors are borderline. Three or more means you should not trust the model for production triage yet.

Scheduling it on a free server

The harness is stateless. It needs no database, no secret storage beyond environment variables. That makes it a natural fit for MonkeyCode's free server option.

Deploy the runner as a cron-like scheduled task:

# Install dependencies
pip install requests

# Run the audit every Monday at 09:00 UTC
0 9 * * 1 python /app/eval_harness.py >> /var/log/eval.log 2>&1

The output is compact. Send it to a private Slack channel or just check it manually on Monday. The free tier covers the weekly token cost easily for this workload. You are not building an app; you are building a monitor for the app.

Where this approach fails

Forty-five examples will not catch every regression. The set is static by design, so it remains useful for measuring drift, not for discovering new failure modes. If the endpoint changes its response format completely, the parser may raise an exception rather than fail gracefully.

You should not use this harness if you cannot define deterministic labels, or if your texts contain sensitive data that cannot be sent to an external endpoint. It is also not a replacement for real human review on high-stakes moderation or medical classification. The harness is a tripwire, not a judge.

Make it a habit

The cost of this safety net is almost zero. One CSV file, one Python script, and a few free-tier API calls per week. The alternative is discovering drift in a user-facing bot after it has already mislabeled a dozen real issues.

Set up the harness now, run it once, and store the baseline. When the endpoint changes, you will see the signal before your users do.