I set out to have a language model classify integration failures. I built an evaluation harness to prove it worked. The harness proved it wasn't worth using.
Final architecture: deterministic rules do the classification and structured extraction. The LLM does exactly one thing, writing plain-English incident summaries. Cheap rules to sort, AI to explain.
Harness is here: https://github.com/fissible/llm-triage-eval
Here are the numbers, then how I got them.
The headline
- Classification: rules 89.7%, LLM 87.9%, across 58 hand-reviewed cases.
- Prompt engineering: v1 = v2 = v3 = 89.7%. Three prompt revisions moved nothing.
- Extraction: against a fair baseline, LLM and regex agree 90 to 97% on structured fields. My earlier claim that extraction was the LLM's territory was wrong.
- Summaries: faithfulness 4.62/5, validated. Completeness 4.26/5, retracted (more on that below).
- Cost: $0. Local model via Ollama, roughly 8 to 10 seconds per case.
The problem
A legacy integration layer is being retired and rebuilt. Its failures are voluminous, repetitive, and hard to reason about: a wall of stack traces where the same handful of root causes repeat thousands of times.
I wanted a failure taxonomy and triage tool. Feed it exported logs, get back a categorized set of distinct problems rather than an undifferentiated pile.
Secondary benefit that turned out to matter: the labeled failure set doubles as a migration-acceptance suite. If the replacement handles all 58 distinct failures correctly, it covers what the old system was actually failing on.
The data pipeline
Roughly 4,000 parsed failures collapse to 58 distinct signatures, deduplicated by error type plus normalized message plus originating element. In one case, 675 expression errors were approximately one bug.
Two things mattered more than anything model-related.
Streaming parse. Line-by-line via a generator, constant memory on 100MB+ files. Header regex plus block-field regex.
Correlated-detail enrichment. This is the key move and it's entirely deterministic. A failing transaction logs a generic error at ERROR level, and separately, at WARN, a JSON blob keyed by correlation ID carrying the actual root cause. A two-pass scan collects the detail blocks by correlation ID and attaches them to the error case.
The impact was large. One generic routing category de-aliased from 287 cases to 85. A database timeout category went from 0 to 60, because the real cause had been invisible behind a wrapper.
That is worth sitting with. The single biggest improvement in classification quality came from stitching correlated log lines together with regex, before any model saw anything.
The taxonomy and the doctrine
Thirteen categories, derived from frequency counts on real logs, defined by root cause rather than error code.
The labeling standard, which I'll call the doctrine: classify by the deepest root cause visible in the correlated details, not the generic wrapper. A routing error whose underlying detail is a duplicate key violation is a constraint violation, not a routing error. With one nuance: a timeout calling a downstream service over HTTP is a downstream timeout; a true database timeout requires the JDBC call to be inside the failing component.
A rule-based classifier pre-tagged every case, which made building the golden set a review task rather than blind labeling. That weak labeler also became the deterministic baseline the LLM had to beat, which turned out to be the most consequential design decision in the project.
The 58 cases are sanitized, environment-tagged, and human-reviewed. The original error type is withheld from the model's input, since including it would trivialize the task.
The harness
Three eval types:
- Classification. Accuracy, per-category precision and recall, confusion matrix, compared against the rule baseline.
- Extraction. Per-field accuracy on HTTP method, status, target entity, error type, operation, and constraint, compared against a regex parser.
- Summary. The model writes a one-sentence incident summary; an LLM-as-judge scores faithfulness and completeness from 1 to 5.
Discipline throughout: temperature 0, structured output via schemas, every run writes a timestamped JSON report, prompts are versioned so regressions are diffable, and the golden set is the regression target.
php artisan triage:eval --source=db --only-reviewed --prompt=v3
php artisan triage:eval-extract
php artisan triage:eval-summary
Findings
Rules beat the model at classification. 89.7% to 87.9%. Not a rout, but the model has to be better to justify latency, cost, and nondeterminism, and it wasn't better at all.
Prompt engineering did nothing. Three prompt versions, identical accuracy. The entire improvement in this project, an 8.7 point jump from 81% to 89.7%, came from making the labeling doctrine coherent. Not from prompts. Once I could state clearly what the correct label was, both the rules and the model improved, and the rules improved more.
The harness caught a bug in itself. Running with the v2 prompt flag produced a report labeled v2, but the runner never threaded the version through to the classifier, so it silently re-ran v1. The tell was identical token counts across supposedly different runs. Without the harness recording token counts per run, three prompt versions would have looked like a real negative result rather than a wiring bug producing a fake one.
A fair baseline erased the model's apparent edge on extraction. With a shallow regex, the model looked dramatically better, because it dug into correlated details the regex ignored. Making the regex details-aware closed the gap: agreement of 96.5% on HTTP method, 94.8% on status, 93.1% on constraint, 89.7% on operation. The two fields that still diverge, error type at 62% and target entity at 83%, differ for definitional reasons rather than accuracy: when a failure chain carries several error codes, which one is "the" type is a question about the taxonomy, not the extractor.
I had previously claimed extraction was where the LLM earned its place. That claim was an artifact of a weak baseline.
Summaries are where the model wins. Given a failure with correlated details, it produces a readable root-cause-and-cascade narrative that no regex is going to write.
Validating the judge
Scoring summaries with an LLM judge that shares a model with the summarizer is self-evaluation, and I didn't want to report those numbers without checking them. So I scored 15 summaries by hand and compared.
| Axis | Exact | Within-1 | Mean bias | Quadratic-weighted κ | Verdict |
|---|---|---|---|---|---|
| Faithfulness | 80% | 100% | -0.07 | 0.57 | Moderate |
| Completeness | 40% | 80% | +0.13 | 0.10 | Do not trust |
Faithfulness holds. Never off by more than one point, near-zero bias, κ of 0.57. The 4.62 stands with moderate confidence. This is the axis that matters most, since it's asking whether the model invented facts.
Completeness does not. κ of 0.10, exact agreement 40%. Don't cite the 4.26. Two things are likely tangled: completeness is genuinely more subjective, and with only 15 items scored mostly 4s and 5s, κ becomes unstable and reads low even where within-1 agreement is 80%. Either way the honest call is inconclusive.
The most useful thing I learned here wasn't statistical. Scoring completeness by hand was hard, and I disagreed with my own earlier judgments. That's not a judge problem, it's a rubric problem. I had written a coherent doctrine for classification labels and never wrote one for what "complete" means in a summary. The judge disagreed with me because I disagreed with myself.
LLM-as-judge worked on the objective axis and failed on the subjective one, which is roughly what you'd hope, and worth knowing rather than assuming.
The resulting architecture
Deterministic rules and parsers do classification and structured extraction. The LLM writes the plain-English summary on demand.
The LLM classifier and extractor still exist, but only inside the eval harness, as the measured contender. Swapping to a stronger model is a two-value config change, so when I want to know whether a frontier model changes the answer, the benchmark is already there. It just isn't in the live path.
What I'd tell you to take from this
- Build the eval harness first. Its job isn't to prove the model works. It's to find out where the model belongs, and the answer might be "almost nowhere."
- A fair baseline is everything. A weak regex made the model look great. A strong one erased the edge. Strengthen the baseline before concluding the model wins.
- Coherent labeling doctrine beats prompt engineering. The entire gain in this project came from knowing what the right answer was. Prompt versions were noise.
- The harness catches the builder's mistakes. It caught my silent prompt-wiring bug and it caught my premature claim about extraction. Both were my errors, and both were invisible without measurement.
- Root cause hides behind generic wrappers. Stitching correlated context together was the highest-leverage work in the project, and it was regex.
- Validate your judge. And when it disagrees with you, consider that your rubric might be the problem.
- Local-first is honest about cost. Free iteration, no data leaving the machine, and a benchmark that can measure hosted models later rather than assuming them.
The uncomfortable version of all this: I spent most of the project building the thing that told me not to build the thing. That was the project working correctly.
The harness is on GitHub: https://github.com/fissible/llm-triage-eval
Top comments (0)