Triage CI Failures with a Free-Tier Model: A Reproducible Classifier Pipeline

A red build stops your team. Someone must read the logs. Someone must decide: flaky, regression, or environment? That decision is mechanical most of the time. A small model can do it. This article builds a failure classifier. It runs on free models and a free server. You get a script you can run today.

The hidden cost of failure triage

Every red CI run interrupts flow. The first responder reads logs, checks the diff, and guesses. Most failures fall into four buckets. The guess is usually right, but it still costs ten minutes. Multiply that by every developer and every broken build.

Automation rarely helps because the tooling is too heavy. Full observability platforms need setup. Custom classifiers need labeled data. A free-tier model needs neither.

Why a small model is enough

Failure triage is a classification task with short inputs. The log is a few hundred lines. The diff is a few files. The output is one label and a short justification. This is not a reasoning-heavy job. A free model handles it well.

MonkeyCode is an open-source project that fits this exact spot. It offers free models and a free server option. The pipeline below runs at zero inference cost. The script uses a generic CLI adapter, so you can swap the backend later.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The decision table

Before writing code, define the labels. Use five buckets. Keep them mutually exclusive.

Label	Signal	Action
flaky	Test failed once, no related diff	Re-run in CI
regression	Diff touches the failing path	Assign to author
environment	Dependency, network, or timeout	Re-run, then escalate
test_bug	Assertion or fixture error	Fix the test
unknown	No clear signal	Manual review

The table is the prompt. The model maps evidence to a label. The evidence field is what you verify.

The classifier script

The script below reads a CI log and a diff. It sends both to the model. It expects a JSON response with three fields: label, confidence, and evidence.

#!/usr/bin/env python3
"""triage.py — classify a CI failure with a free-tier model."""
import json
import subprocess
import sys
from pathlib import Path

LOG_PATH = Path(sys.argv[1]) if len(sys.argv) > 1 else Path("ci.log")
DIFF_PATH = Path(sys.argv[2]) if len(sys.argv) > 2 else Path("diff.txt")

LABELS = {
    "flaky": "Test failed once, no related diff.",
    "regression": "Diff touches the failing path.",
    "environment": "Dependency, network, or timeout.",
    "test_bug": "Assertion or fixture error.",
    "unknown": "No clear signal.",
}

PROMPT = f"""
You are a CI failure triage assistant.
Read the log and the diff. Choose exactly one label.
Labels: {json.dumps(LABELS)}
Return JSON: {{"label": "...", "confidence": 0.0-1.0, "evidence": "..."}}
Evidence must quote the log. Do not invent facts.
"""

def run_cli(prompt: str) -> str:
    """Send a prompt to the model CLI via stdin and return stdout."""
    result = subprocess.run(
        ["mc", "--free-server", "prompt"],
        input=prompt,
        capture_output=True,
        text=True,
        check=True,
    )
    return result.stdout

def main() -> None:
    log = LOG_PATH.read_text()[-4000:]
    diff = DIFF_PATH.read_text()[-2000:]
    response = run_cli(PROMPT + f"\n\nLOG:\n{log}\n\nDIFF:\n{diff}")
    try:
        data = json.loads(response)
    except json.JSONDecodeError:
        print("{\"label\": \"unknown\", \"confidence\": 0.0, \"evidence\": \"Model output was not JSON.\"}")
        return
    print(json.dumps(data, indent=2))

if __name__ == "__main__":
    main()

The script truncates inputs. Long logs lose context, but the tail usually contains the failure. The diff tail contains the most recent changes.

Prompt design notes

The prompt has three deliberate constraints. First, it forces one label. A model that can answer "unknown" will overuse it. Second, it demands quoted evidence. That makes the output verifiable. Third, it forbids invented facts. That keeps the model honest.

You can tune the labels later. Some teams split regression into logic and data. Some add a migration label. The structure stays the same: label, confidence, evidence.

Run it locally first

Before touching CI, run the script on a saved failure. The command is simple:

python triage.py ci.log diff.txt

You should see JSON on stdout. If the model returns prose instead, add a system prompt that demands JSON. If the output is still malformed, fall back to the unknown label and review manually.

Example output

A healthy run returns clean JSON. Here is a realistic response:

{
  "label": "regression",
  "confidence": 0.87,
  "evidence": "test_auth.py:41 failed; diff changes validate_token() signature."
}

The evidence is the part you trust. The label is a hypothesis. The confidence is a hint, not a guarantee.

A second run might return a different verdict:

{
  "label": "flaky",
  "confidence": 0.74,
  "evidence": "test_orders.py:22 failed; diff touches billing.py only."
}

That is the case where the model earns its place. It reads the diff, finds no connection, and says so. A tired developer might have missed that.

Run it in CI

Add the script to your pipeline. Use three numbered steps.

1. Capture the failing log and the diff before the job exits.
2. Run the classifier and write the JSON to a file.
3. Post the label and evidence as a comment on the commit.

A minimal CI snippet looks like this:

- name: Capture failure
  if: failure()
  run: |
    tee ci.log
    git diff origin/main...HEAD > diff.txt
- name: Classify failure
  if: failure()
  run: python triage.py ci.log diff.txt > triage.json
- name: Comment result
  if: failure()
  run: gh pr comment "$PR" --body "$(cat triage.json)"

The comment gives the next developer a starting point. It does not replace judgment. It removes the first ten minutes of reading.

Validate the buckets first

Do not wire this into CI immediately. Run it on last week's failures first. Compare the model labels with what your team actually decided. Track three numbers: precision on regression, precision on flaky, and the unknown rate.

Metric	Target
Regression precision	> 0.8
Flaky precision	> 0.7
Unknown rate	< 0.3

If the unknown rate is above 30 percent, your buckets do not match your logs. Adjust the labels and try again.

Validation is cheap. It costs one script run and a few minutes of reading. It prevents the worst outcome: a confident model that is confidently wrong.

When the classifier lies

The model will be wrong. Treat the label as a suggestion. The evidence field is the real deliverable. It points at the log lines that matter.

Do not use this for security incidents. Do not use it for on-call pages. Do not let it auto-close issues. The pipeline is a triage aid, not an authority.

Who should not use this

Teams without a human review step will trust the label blindly. That is worse than no automation. Regulated environments need a documented decision trail, not a model summary. If your CI logs contain secrets, do not send them to any remote model. The free server option changes cost, not data handling.

Try it with zero cost

The full pipeline costs nothing to run. MonkeyCode's free models and free server cover the inference. The script is a template. Adapt it to your CI system and your failure patterns.

Start with last week's failures. Run the classifier on old logs. Compare its labels with what your team decided. That validation tells you whether the buckets fit your reality. If they do, wire it into CI and reclaim the first ten minutes of every red build.