I closed an important stage of ops-triage-ai, an operational triage
system that combines a deterministic baseline, a local LLM, and a hybrid
policy with human review.
The most interesting result was not simply "the LLM was better."
What the benchmark measured
Evaluation on a frozen held-out set of 70 synthetic tickets, in a single
run. Deterministic baseline versus the local LLM:
- Category accuracy: 82.9% → 95.7%
- HIGH/CRITICAL priority recall: 78.6% → 100%
- HIGH risk recall: 57.1% → 71.4%
And the point that matters: the deterministic baseline still won on
overall risk accuracy — 95.7% against the LLM's 91.4%.
The LLM did not win everywhere. The regression is published alongside the
gains, with nothing smoothed over.
The question changed
Instead of:
"How do we replace rules with AI?"
the architecture answers:
"How do we combine different behaviors in a way that is safe, observable,
and auditable?"
The final architecture
- Deterministic baseline as reference and fallback;
- Local LLM for semantic interpretation;
- Hybrid policy to decide when human review is required;
- Audit trail preserving ticket, run, decision, and feedback.
In the final benchmark:
- 51.4% of cases were routed to human review;
- the classifiers disagreed on 32.9% of tickets;
- there was no fallback and no LLM failure during the official run.
Review rate describes operations here — how many cases asked for review —
not review precision/recall. The held-out set has no independent ground
truth for that.
Accepted limits
Synthetic dataset, in English, with 70 examples. A single official run,
with no measurement of LLM variance. The system has not been validated
under real production traffic. Hybrid latency sits around 6 to 7 seconds:
fine for asynchronous triage, not for a critical synchronous path. The
full details — official metrics, trade-offs, and the 13 published
limitations — are in the project case.
The takeaway
A useful AI system does not need to trust the model blindly.
Sometimes the best architecture comes precisely from understanding where
the model is better — and where it is not.
Top comments (0)