48-Hour Field Notes: Let the Triage Bot Say 'I Don't Know'

A pipeline fails at 02:40. Someone pastes the log into a chat window, reads the answer, and types a summary back into the incident channel. On its own, that is fine. Do it forty times a week and the team has invented a job whose only function is to relay bytes between two systems that could talk to each other directly.

The 48 hours below are about deleting that relay job without replacing it with a worse one: an auto-labeler that guesses. The rule I set for the exercise was simple. The bot is allowed to classify a failure only when it can point at the substring that decided the answer. Everything else goes to a human, on purpose, with no penalty attached.

The setup I actually ran

I split a folder of pasted CI excerpts into two files: gold.jsonl (human labels, kept out of reach of the prompt) and excerpts.jsonl (raw text, truncated to the last 120 lines, because the tail is where the failure usually lives). Then I wrote one process that answers and one that scores, and let the process run in shadow mode for two days: it logged every decision and acted on none of them.

For the two days I used MonkeyCode's free model access to serve the classification calls, and put the logging service on the free server option so the test would survive a laptop lid closing. As advertised at the time of writing, that means free model access plus a free server, with roughly 10 million free tokens on offer. I have not verified current model names, quotas, server specs, or how long any of it lasts, and neither should you take my word for it.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The important property of a free allowance here is not its size. It is that a retry loop bug cannot turn into an invoice while you are asleep, which is exactly the failure mode a two-day unattended shadow run invites.

Rules first, model second, human last

The code below is a reference implementation, not a transcript of a production run. The call_model function is a placeholder; point it at whichever endpoint you hold credentials for. What matters is the shape: cheap deterministic checks buy down the model's job, and the model is only trusted when its answer is grounded in the text it was given.

import json, re

TAXONOMY = ('dependency', 'credentials', 'infra', 'flaky_test', 'noise')

RULES = (
    ('dependency',  re.compile(r'ModuleNotFoundError|Cannot find module|no matching distribution', re.I)),
    ('credentials', re.compile(r'authentication failed|401 Unauthorized|permission denied', re.I)),
    ('infra',       re.compile(r'OOMKilled|exit code 137|connection refused|Timeout after', re.I)),
)

PROMPT = '''You classify exactly one CI failure excerpt.
Allowed labels: {labels}
Reply with JSON only: {{"label": "<one allowed label>", "span": "<smallest substring from the excerpt that decided it>"}}
If the excerpt does not clearly match one label, use "noise".
Excerpt:
---
{excerpt}
---
'''

def classify(excerpt, call_model, min_span=8):
    for label, pattern in RULES:
        m = pattern.search(excerpt)
        if m:
            return {'label': label, 'span': m.group(0), 'mode': 'rule'}

    raw = call_model(PROMPT.format(labels=', '.join(TAXONOMY), excerpt=excerpt))
    try:
        parsed = json.loads(raw)
    except json.JSONDecodeError:
        return {'label': None, 'span': None, 'mode': 'escalated', 'why': 'unparseable'}

    label, span = parsed.get('label'), (parsed.get('span') or '').strip()
    if label not in TAXONOMY:
        return {'label': None, 'span': span, 'mode': 'escalated', 'why': 'off_taxonomy'}
    if label == 'noise' or len(span) < min_span or span not in excerpt:
        return {'label': None, 'span': span, 'mode': 'escalated', 'why': 'ungrounded'}
    return {'label': label, 'span': span, 'mode': 'model'}

The span not in excerpt check does more work than any prompt engineering I tried. It converts a confident wrong answer into a refusal, and a refusal has a known cost: someone reads the log, which is what they were already doing.

Signal	Action	Why
Regex hit, span present	Auto-label, no model call	Deterministic, free, auditable
Model label, span found in excerpt	Auto-label in shadow; promote only after review	Grounded, still fallible
Model label, span missing or too short	Escalate	Confabulation signal
Label outside taxonomy, unparseable JSON	Escalate and log raw output	Prompt or parser drift
noise	Escalate	Absence of evidence is not a label

Shadow mode is the test plan

A classifier that acts immediately cannot be evaluated, because you never see what it would have done on the cases it got wrong. Two days of shadow mode fixes that. The service writes one line per decision to decisions.jsonl with id, label, mode, and the token counts returned by the API; nothing is posted anywhere.

import json
from collections import Counter

def load(path):
    with open(path) as fh:
        return [json.loads(line) for line in fh if line.strip()]

def score(decisions, gold):
    gold = {r['id']: r['label'] for r in gold}
    total = len(decisions)
    auto = [d for d in decisions if d['mode'] != 'escalated' and d['id'] in gold]
    correct = sum(1 for d in auto if d['label'] == gold[d['id']])
    return {
        'total': total,
        'coverage': round(len(auto) / total, 3),
        'precision_on_auto': round(correct / len(auto), 3) if auto else None,
        'escalation_rate': round(1 - len(auto) / total, 3),
        'silently_wrong': len(auto) - correct,
        'by_mode': dict(Counter(d['mode'] for d in decisions)),
    }

silently_wrong is the only number that should gate promotion. Coverage can look great while that field climbs; a high escalation rate is not a failure, it is the system telling you where the taxonomy is thin.

What broke

JSON parsing broke first. A model that adds a friendly sentence before the object is not doing anything wrong, but it does mean your parser needs a strict-repair path rather than a json.loads and a prayer. I now strip to the outermost braces before parsing and count repairs as a metric.

Grounding broke second, in the quiet direction. Short spans like timeout appear everywhere, so a length floor of eight characters is doing real work. The failure I did not expect was label drift: the model started using words from the log itself ("segfault", "deadlock") as labels, which look plausible and are outside any taxonomy you can score against.

Truncation broke third. The last 120 lines are right for most failures and wrong for the ones where the real error is at line 400 and the tail is a wall of retry noise. The fix was boring: if the excerpt contains no line matching error|fail|fatal, escalate early instead of asking the model to be brave.

Cost, as arithmetic rather than a promise

Measure your own tokens before you plan anything. In my template, one call is about 200 input tokens and around 30 output tokens, so call it 230 per classification, and the free allowance of 10,000,000 divides out to roughly 43,000 calls. That is arithmetic on my prompt template, not a product claim, and it assumes no retries and no caching. Divide by your p95 excerpt length instead of your p50 if you want a number that survives contact with a bad week.

The cheaper saving is upstream. Every regex hit is a call you never make, and in a CI corpus the dependency and credential families are boringly regular.

Who should not do this

Do not run this on decisions with legal, medical, or safety consequences; an abstain path does not make a wrong label acceptable there. Do not run it if you have no labeled set at all, because shadow mode without gold labels only measures activity. Do not run it if nobody owns the escalation queue, since the design deliberately creates work for a human and that work has to have an owner. And do not run it on logs containing secrets or customer data until you have decided what the model provider is allowed to see, which is a policy question before it is a code question.

What I'd repeat

The regex-first layer, the span-containment check, and the two-day shadow window are the three things I would carry into the next project unchanged. They are all cheap, and together they turn a confident classifier into a cautious one. If you are wiring a model into an alerting path this week, the order that worked for me was: refusal first, coverage second, model access last.