A Flake Triage Card for Your CI: Classify Failures First, Read Logs Second

CI goes red for reasons that have nothing to do with your change. A port collision on a shared runner, a test that expects a clock to move, a missing dependency image, a stale cache, an assertion that finally caught a real regression. When every failure lands as a raw stack trace in a channel, the expensive part is not the failure; it is deciding which log to open. A small classification pass can sort the queue before any human reads a single line.

The script below uses free model access and a free server option from MonkeyCode, so you can run it on a scheduled job without adding infrastructure. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

A triage card, not a root-cause verdict

The goal is not to explain the failure. The goal is to produce a triage card with three fields that help you decide what to read next: category, confidence, and suggested_next_step.

Use these categories to start:

• infra: network, port, timeout, disk, memory, permission, connection refused
• code: import error, syntax error, missing module, config parse, build failure
• flake: race, order dependence, retry works, timing or clock sensitivity
• test_logic: assertion mismatch, expected value changed, invalid fixture
• ambiguous: truncated log, generic error, no stack trace, unrelated noise

A model proposing one of these labels is a draft, not a decision. The script merges that draft with a few deterministic hints from keywords you already know, then prints a card you can scan before opening the full log.

Reproducible triage script

The script takes a directory of failed CI log snippets, calls the model on the last 4,000 characters of each file, and adds regex hints that can only escalate attention, never downgrade it. Replace the classify_log stub with the endpoint you have access to.

# flake_triage.py
import json
import re
from dataclasses import dataclass
from pathlib import Path

CATEGORIES = {
    "infra": "network, port, timeout, memory, permission, connection refused",
    "code": "import error, syntax error, missing module, config parse, build failure",
    "flake": "race, order dependence, retry works, timing, clock, sleep",
    "test_logic": "assertion mismatch, expected value changed, invalid input fixture",
    "ambiguous": "log truncated, unrelated failure, no stack, generic error",
}

PROMPT = """Classify this CI failure log excerpt.
Categories: infra, code, flake, test_logic, ambiguous.
Return JSON with fields:
category, confidence (0-1), reason (one sentence), suggested_next_step (one sentence).
Do not claim a specific root cause beyond the log.
LOG:
{log}
"""

def classify_log(log: str) -> dict:
    # Replace with the model endpoint available to you.
    raise NotImplementedError

def rule_hints(log: str) -> dict:
    hints = []
    if re.search(r"connection refused|timed out|address already in use", log, re.I):
        hints.append("infra")
    if re.search(r"ImportError|ModuleNotFoundError|SyntaxError", log):
        hints.append("code")
    if re.search(r"flaky|retry|race|deadlock|eventually", log, re.I):
        hints.append("flake")
    return {"rule_hints": hints}

def triage_one(failure_blob: str) -> dict:
    model = classify_log(failure_blob)
    hints = rule_hints(failure_blob)
    # Model output is a draft; rule hints can add a competing hypothesis,
    # but they never override a lower-confidence label.
    if model.get("confidence", 0) < 0.6:
        model["suggested_next_step"] = "read the full log before deciding"
    return {"model": model, "rule_hints": hints}

def main(log_dir: Path):
    logs = sorted(log_dir.glob("*.log"))
    for log_path in logs:
        text = log_path.read_text(errors="ignore")[-4000:]
        result = triage_one(text)
        m = result["model"]
        print(f"{log_path.name}\t{m.get('category')}\t{m.get('confidence')}")
        print(f"  reason: {m.get('reason')}")
        print(f"  next:   {m.get('suggested_next_step')}")
        if m.get("confidence", 0) < 0.5 or result["rule_hints"]:
            print(f"  hints:  {result['rule_hints']}")

if __name__ == "__main__":
    main(Path("ci_logs"))

The rule_hints function exists for the cases where you know the word patterns that keep recurring in your environment. If a log says address already in use, the model might call it infra with high confidence or ambiguous with low confidence. Either way, the hint stays visible so the human reading the card can challenge the label.

Reading the card

Not every model label deserves the same response. Use this decision table to turn the card into an action:

Model category	Confidence	Action
infra	high	Check runner, network, or port once, then rerun that job
code	high	Inspect import/build error before rerunning
code	low	Read the full log; do not just rerun
flake	any	Run the failing test three times; if 2 of 3 fail, treat as code
test_logic	high	Compare the diff to the fixture or assertion; likely intended change
ambiguous	any	Read the full log; no automated routing is safe

The table is deliberately conservative. A model saying flake is not permission to close an issue as flaky. It is a prompt to test determinism before assigning blame.

Calibrate it against your last twenty failures

Take twenty closed CI failures where you already know the true cause. Run the script against the saved logs and compare the proposed category with your ground truth. Count precision per category, not just overall accuracy. You will often find that infra and code are easy because the signatures are strong, while flake and test_logic are noisy. That local number is more useful than any external benchmark because it reflects your stack, runner images, and test suite habits.

When you swap model endpoints, rerun the same calibration set. Different instruction-tuned models will disagree on confidence even when they agree on category, and the confidence threshold you learned from one endpoint may not transfer cleanly to another.

Limitations

• Tail logs are lossy. A failure whose cause appears in the first 500 lines may look like ambiguous if you only pass the last 4,000 characters.
• Categories overlap. A race condition can surface as a connection refused, and a missing dependency can look like a test logic failure. The card is for ordering attention, not for claiming a root cause.
• The script does not access your test code or diff by default. That is intentional: it should stay cheap and low-context. If you want better labels, add only the failing test name and the diff summary, not the entire repository.
• Logs may contain environment names, file paths, and secrets. Sending them to a third-party model endpoint is a data-handling decision that requires the same review as sharing any other CI artifact.
• Do not auto-close, auto-retry, or auto-assign based on the model label. The only automatic part here is the sorting of a list for human review.

Who should skip this workflow

• Teams with compliance or privacy rules that forbid sending CI logs to external model endpoints.
• Small teams where every red build is already read by the same person; adding classification would be ceremony instead of leverage.
• Repositories with very stable CI where flake is rare and failure categories are obvious from file names alone.
• Teams expecting fully automated root-cause analysis or auto-remediation in one step. This workflow gives you a reading order, not a fix.

Start with five recent failed jobs and a one-line summary for each. The point is not fewer logs; it is a better order in which to read them. If you already have free model access through MonkeyCode, point the script at the CI artifacts on a free server and let it draft the first pass; a human still owns the decision.

Comments

[DaymondHyper]

Thanks for writing this, useful and practical. The point about keeping it simple is the one that resonates most with me. Have you found a setup that works well for you so far?