When OpenAI launched the Decisions API on Luna this week, and TypeSafe launched Jev two weeks before that, they were both making the same bet: the future of AI in production is not bigger models, it is smaller, faster decision models embedded in your workflow.
I agree. And I have the numbers to prove it works.
laya-triage routes support tickets through a local decision model: intent (77 categories), department (8), urgency, frustration, churn risk, refund flag, and a confidence-gated escalation to a human. All in one forward pass, on CPU, for free.
Hierarchical intent accuracy: 51.0% -> 90.5% (after fine-tuning)
Coarse-cluster accuracy: 68.0% -> 96.0%
Urgency MAE: 0.81 -> 0.70
Cost per ticket: $0
The architecture
Flat 77-way classification is hard for a small model. So I broke it into two stages:
Ticket -> [coarse stage: 12 clusters] -> [fine stage: 3-10 candidate intents] -> decision
|
+-> urgency, frustration, churn, refund (same forward pass)
|
+-> confidence check -> escalate to human if below 0.84
The hierarchy alone added 14.5 percentage points over flat routing. The fine-tune added another 39.5.
What I fine-tuned on
10,003 BANKING77 tickets, split into two sequences per ticket, trained on Kaggle's free 2x T4 GPUs. The checkpoint is published on HuggingFace: Gjusev/laya-triage-banking77.
The eval harness that tracked the improvement ran on Z.ai's GLM models, with GLM-5.3-flashX as the workhorse. When the judge model changes, the numbers move a little. The fine-tune gain does not.
The results, before and after
| Measurement | Zero-shot | Fine-tuned |
|---|---|---|
| Hierarchical intent accuracy | 51.0% | 90.5% |
| Macro-F1 | not measured | 0.848 |
| Coarse-cluster accuracy | 68.0% | 96.0% |
| Urgency MAE | 0.81 | 0.70 |
| Frustration MAE | 1.07 | 1.07 (untouched heads) |
At the 0.84 confidence threshold (zero-shot): 75.2% accuracy with 60.5% of tickets auto-handled. The rest escalate to a human with a recorded reason.
The escalation policy (and why it matters)
The threshold gates only on cluster and intent confidence. Signal confidence (urgency, frustration, churn) is deliberately not used, because score-type questions have structurally lower confidence and would trigger near-universal handoffs.
On an 8-ticket out-of-domain smoke test (IT-ops tickets against banking intents): 7 escalated correctly. The one that did not, an SSO lockout, mapped to "unable to verify identity". Not wrong, exactly, but the reason it matters is that the behavior is visible and fixable, not hidden behind an opaque binary.
Try it
The demo is live on Streamlit, in 6 languages (English, Spanish, French, German, Hindi, Arabic):
Or run it yourself:
pip install laya-triage
Fine-tuning notebook included for Kaggle (2x T4, ~4 hours).
The rest of the series
- laya-router: route prompts between cheap and frontier models, 54.9% cost reduction
- laya-compactor: cut 70% of RAG context tokens, same answer quality
- laya-phishield: explainable phishing detection, $0 per 1,000 emails
All four tools use the same laya decision engine, Apache 2.0, local, with published benchmarks. If OpenAI's Decisions API and TypeSafe's Jev are the proprietary versions of this idea, this series is the open one. Run it on your machine, keep your data, pay nothing per decision.
Top comments (6)
Which split is the 90.5% on? 10,003 is exactly BANKING77's training set, and the dataset ships a fixed test split of 3,080 queries. If the before/after numbers come from that official test split, they're directly comparable with the published fine-tuned baselines on the same dataset, which is the most convincing way to show the hierarchy helps. If they come from a sample of the training tickets, some of the gain may be memorisation.
The round figures (51.0, 68.0, 90.5, 96.0) also look like an evaluation set of about 200 tickets. At that size, 90.5% has a 95% interval of roughly 86% to 94%, which is fine for the 39-point jump but too wide for the 14.5-point hierarchy-vs-flat comparison unless both were scored on the same tickets and compared pairwise. Stating the n and the split next to the table would settle both questions.
On the split: every number in the post is from the test split. I evaluate on a fixed sample of 200 tickets (seed 13) drawn from the 3,080 test queries, and it's the same sample for every configuration. The 10,003 figure you spotted is the train split, which only the fine-tune fitted on. The fine-tuned model never saw any of the eval tickets, so memorization isn't in play. You're right that the post didn't say this next to the numbers. It does now.
On n = 200: also right, and worth spelling out. The independent intervals are too wide to settle the 14.5-point gap on their own; the 90.5% alone spans roughly 86 to 94%. But the comparison is paired. All configurations scored the same tickets and the per-ticket records are in the repo, so I can count directly: of the 73 tickets where flat and hierarchical disagree, the hierarchy wins 51 and loses 22. Exact McNemar gives p = 0.0009. Against the fine-tuned model it's more lopsided, 81 to 2 and 109 to 1. I added this as a section of the results report (github.com/Gjusev/laya-triage/blob...) and updated the post.
The 200-ticket sample was a compute-budget choice for the CPU zero-shot runs. Scoring all three configurations on the full 3,080-ticket test split is the natural next step for direct comparison with published baselines. The harness already supports --full; I'll add those numbers when the run finishes.
51 vs 22 checks out: I get the same exact two-sided p of 0.0009. The paired numbers also give you the size of the gap, not just its sign: (51 - 22) / 200 = 14.5 points, with a 95% interval of roughly 6 to 23 points. So the hierarchy clearly helps, but on 200 tickets "somewhere between 6 and 23 points" is as precise as it gets. If the disagreement rate stays near a third, the full 3,080 run should narrow that to about ±2.
One thing to look at while it runs: the 73 disagreements broken down by intent. BANKING77 has a few near-duplicate intents (the card_arrival / card_delivery_estimate kind). If most of the hierarchy's 51 wins sit in those pairs, the gain comes from the coarse-to-fine step separating confusable labels, which is a more useful finding than one overall number.
Confirmed on the p-value, and the 6 to 23 point interval is the honest summary at n = 200. Full 3,080 run is going now.
I ran the intent breakdown from the committed records. Short version: no single pair dominates. The 51 wins spread over 27 gold intents, the biggest pair is 3 tickets. Splitting by mechanism, 21 wins had flat's wrong answer inside the correct cluster (your disambiguation case), but 30 had flat leaving the cluster entirely, so most of the gain is the coarse step recovering misroutes, not fine-grained separation. The 22 losses do cluster a bit: passcode_forgotten and contactless_not_working three times each.
I'll post the full-split numbers with this breakdown at n = 3,080 when the run lands.
Full split is in, and your ±2 estimate was about right. All 3,080 test tickets, same three configurations: flat 37.4%, hierarchical 51.1%, fine-tuned 89.8% (macro-F1 0.897). Every n = 200 number replicates within a point. Paired gap flat vs hierarchical: +13.7 pp, 95% CI roughly +11.4 to +16.0, discordant tickets 821-399.
Reran the intent breakdown at full size, and your prediction holds in an interesting way. The 821 wins spread over 66 gold intents, no single pair dominates. 39% are the case you described, flat's wrong answer inside the correct cluster, and the near-duplicate pairs you guessed do show up there: unable_to_verify_identity vs verify_my_identity (22), lost_or_stolen_card vs lost_or_stolen_phone (21), visa_or_mastercard vs supported_cards_and_currencies (21). The other 61% are tickets flat had routed out of the correct cluster entirely, mostly the top-up family, which flat collapses into top_up_by_card_charge from three different gold intents (80 tickets). So the hierarchy helps two ways, and routing recovery is the bigger one.
The losses concentrate on the hierarchy's own confusions: contactless_not_working (33), country_support (30), passcode_forgotten (28, predicted as pin_blocked or change_pin). All in the report now: github.com/Gjusev/laya-triage/blob...
That's a satisfying replication: every n = 200 number within a point, and the interval landing where it should. The two-mechanism split is the real finding, and I'd lead with it.
The losses suggest the next cheap win. If contactless_not_working, country_support and passcode_forgotten fail because the coarse step commits to the wrong cluster, the fine step never gets a chance to fix it. One quick check on the 399 losses: how often was the right cluster the coarse step's second choice? If it's most of them, letting the fine classifier choose among the intents of the top two clusters whenever the coarse margin is small should keep most of the 821 wins and recover part of the 399, at the cost of one extra fine call on uncertain tickets only. Choose the margin cutoff on the training split so the 3,080 stays clean.