Every Monday I used to open my CI dashboard and do the same dreary thing: click into fifteen red pipelines, scroll through thousands of lines of Maven and pytest output, and decide — by vibe — which failures were real regressions and which were the same flaky tests that have been lying to us since March.
The decision itself takes seconds once you see the failure. The expensive part is getting to the failure: finding the actual error line buried under retries, warnings, and teardown noise. So I built a small triage pipeline that does the boring 90% — fetching logs, extracting the signal, clustering near-duplicates — and then asks a language model to write a one-paragraph hypothesis per cluster. The human (me) only reads the hypotheses.
This article is the pipeline: the log-extraction script, the clustering heuristic, the prompt template, and the honest list of places where it falls apart. It runs comfortably on free-tier inference, which is the only reason I bother mentioning the specific tooling I used.
What the pipeline actually does
CI API → raw logs → signal extractor → embed/cluster → LLM summary → triage.md
Nothing here is magic. The design principle is that the LLM is the last stage and the least trusted one. Everything before it is deterministic and cheap, so the model only ever sees a few hundred lines of pre-digested evidence instead of a 40 MB log dump.
Stage 1: Extract signal, not logs
The single biggest mistake people make with "AI + CI" is piping raw logs into a prompt. It blows the context window, costs tokens, and — worse — the model anchors on irrelevant noise like download progress bars. My extractor keeps only lines that match failure-shaped patterns, plus a small window of context around them:
import re
from dataclasses import dataclass
FAIL_PATTERNS = [
re.compile(r"\b(FAIL|FAILED|ERROR|AssertionError|TimeoutError|panic:|SEVERE)\b"),
re.compile(r"^\s*at\s+[\w.$]+\([\w.]+:\d+\)"), # stack frames
re.compile(r"expected.*but (was|got)", re.IGNORECASE),
]
@dataclass
class FailureSnippet:
job_id: str
test_name: str
lines: list[str]
def extract(job_id: str, raw_log: str, context: int = 3) -> list[FailureSnippet]:
lines = raw_log.splitlines()
hits = [i for i, ln in enumerate(lines) if any(p.search(ln) for p in FAIL_PATTERNS)]
if not hits:
return []
# Merge hit windows that overlap, cap total size
windows = []
for i in hits:
lo, hi = max(0, i - context), min(len(lines), i + context + 1)
if windows and lo <= windows[-1][1]:
windows[-1] = (windows[-1][0], max(windows[-1][1], hi))
else:
windows.append((lo, hi))
merged = []
for lo, hi in windows[:20]: # hard cap: 20 windows per job
merged.append("\n".join(lines[lo:hi]))
test_name = next((ln.strip() for ln in lines if "Running" in ln or "::" in ln), "unknown")
return [FailureSnippet(job_id, test_name, merged)]
In my repos this shrinks a typical failing job from ~8,000 lines to ~150. That number matters: it's what makes free-tier inference viable, because you're sending maybe 2 KB per failure instead of half a megabyte.
Stage 2: Cluster before you summarize
If twelve jobs failed on the same flaky test, I want one hypothesis, not twelve. I cluster with a dead-simple heuristic — normalize the extracted snippet (strip timestamps, paths, hex IDs), then group by a similarity threshold on token overlap:
import hashlib
def normalize(snippet: str) -> str:
s = re.sub(r"\d{4}-\d{2}-\d{2}[T ]\d{2}:\d{2}:\d{2}[^ ]*", "", snippet)
s = re.sub(r"0x[0-9a-fA-F]+", "0x…", s)
s = re.sub(r"/[\w./-]+/", "/…/", s)
return s
def cluster_key(snippet: str) -> str:
toks = set(normalize(snippet).split())
# Order-independent fingerprint of the "content words"
sig = hashlib.sha1(" ".join(sorted(toks)).encode()).hexdigest()[:12]
return sig
This is deliberately crude. Exact-fingerprint clustering misses near-duplicates; a fancier approach (embeddings + cosine similarity) catches more but adds a dependency. For my volume — tens of failures a week, not thousands — the crude version catches exact repeats, which are the majority of flake noise. Your mileage will scale with how chaotic your stack traces are.
Stage 3: The model gets evidence, not logs
Each cluster produces one prompt with a tight template:
You are triaging a CI failure. Below are the extracted error lines from {n}
jobs that failed with the same signature, followed by the test name and the
files changed in the triggering merge request.
Answer in exactly three short paragraphs:
1. What most likely broke (code regression vs. test flake vs. infrastructure).
2. The single piece of evidence that most supports that classification.
3. What a human should check first.
If the evidence is insufficient, say so instead of guessing.
---
{snippets}
---
Changed files: {changed_files}
Two details do the heavy lifting. First, the forced three-paragraph structure makes outputs scannable and comparable across failures — free-form "analyze this log" prompts produce rambling essays. Second, the explicit permission to say "insufficient evidence" measurably reduced confident-but-wrong answers in my testing. The failure mode I care about isn't the model being wrong; it's the model being plausibly wrong, because that's what wastes my time.
For the inference side I used MonkeyCode, which currently offers free model access and a free server option — enough headroom for a workload like this that runs a handful of small prompts a day. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Practically, the pipeline is an OpenAI-compatible HTTP call, so swapping providers is a base-URL change; nothing above is locked to any vendor, and you should keep it that way.
What a morning looks like now
The output is a single triage.md I read with coffee:
## Cluster a91f (11 jobs) — checkout_flow_test.rb
1. Test flake. Failure is a Capybara timeout on the payment iframe...
2. Evidence: identical 'expected to find css' line across all 11 jobs;
changed files touch only README and CI config...
3. Check first: whether the flaky-test quarantine list already covers it.
## Cluster 03be (2 jobs) — invoice_service.py
1. Likely code regression: AssertionError on tax rounding after the
decimal migration in this MR...
Reading that takes two minutes. The second cluster was a real regression I would have previously found around lunchtime.
Decision table: when this pipeline helps
| Situation | Worth it? | Why |
|---|---|---|
| Same flaky tests fail repeatedly | Yes | Clustering collapses them to one line |
| Many parallel jobs per pipeline | Yes | Log volume is what makes manual triage slow |
| Green-mostly CI, rare failures | Marginal | Just read the log; setup cost > savings |
| Failures are mostly infra (runner OOM, network) | Partially | Model classifies these well, but you still fix them by hand |
| Regulated/secret-heavy logs | Careful | You are sending log excerpts to an external API — see below |
| You need root-cause proof | No | This produces hypotheses, not verdicts |
Limitations, honestly
- The model has never seen your codebase. It reasons from stack traces and file names. When the changed-files list is short and the error is local, that's enough. When the bug is an emergent interaction between services, the hypothesis is usually a politely worded shrug. Treat output as a triage ordering, not a diagnosis.
- Log privacy is your problem. Extracted snippets can contain usernames, internal hostnames, or data from test fixtures. My extractor strips IPs and tokens with an extra regex pass; if your logs are sensitive, redact before anything leaves your network, or don't use hosted inference at all.
- Free tiers are a gift, not a contract. Quotas, available models, and the server option can change. The pipeline degrades gracefully — if the LLM call fails, I still get the clustered snippets, which is 70% of the value. Design yours the same way.
- Crude clustering misses things. Two manifestations of the same root cause (say, a shared helper throwing different errors) land in separate clusters. I accept this; deduplicating hypotheses matters more than perfect grouping.
Who should skip this
If your CI fails a few times a month, the hour you'd spend wiring this up will never pay back. If your team already has good flake quarantining and ownership rules, the human process is fine and this adds a layer you don't need. And if your logs can't leave your perimeter, hosted free inference is off the table by definition — the extractor and clustering stages still work standalone, but that's a different article.
The actual lesson
The model is the smallest and most replaceable part of this system. The wins came from unglamorous engineering: shrinking logs to signal, deduplicating before summarizing, forcing a rigid output shape, and giving the model permission to abstain. If you build nothing else from this post, build the extractor — it pays for itself even if a human reads its output.
If you want to try the full loop without spending anything, a free model endpoint such as MonkeyCode's is a reasonable place to point the one HTTP call at the end. Start with the triage.md output as a draft you edit, not a report you trust — the day it saves you an hour of log-scrolling, you'll know it's working.
Top comments (1)
The useful part is turning CI logs into a triage artifact, not pretending nobody has to inspect failures. I would keep the raw log linked, the suspected class explicit, and the confidence score visible. Otherwise the summary becomes another thing to debug.