My classifier calls an LLM on every single email. The LLM is not allowed to classify the email.
That sounds like a contradiction. It's the most important design decision in the thing.
A reader named @nazar_boyko left a comment on my last post — the one where a cheap model beat GPT-4o on email triage — and put it better than I did:
Once the LLM is a feature scorer and not the decider, "consistency over genius" falls right out of it, and a cheap fast model is exactly what you want for reading the same four signals the same way every time.
The price upset was the fun headline. This is the actual thesis. So here it is on its own.
The model scores four numbers. That's all it does.
Every inbound email goes to the LLM with one job: read the message and return four scores between 0 and 1.
- confidence — how sure you are the other three scores are right
- senderTrust — 1.0 a known, important human; 0.3 an automated transactional notice you signed up for; 0.0 anonymous bulk marketing
- reversibility — if this got auto-handled and that was wrong, how easy is the recovery? 1.0 trivial undo; 0.0 irreversible ("lost an investor")
- urgency — needs attention within hours (1.0) down to informational, no clock (0.0)
The response schema is literally:
{"confidence":0.0,"senderTrust":0.0,"reversibility":0.0,"urgency":0.0,"reason":"short phrase"}
No tier. The model never sees the words PUSH, QUEUE, SILENT, AUTO in its output contract. It reads an email and describes it along four axes. It does not get a vote on what happens next.
A rule I can read decides the tier
What happens next lives in one file, tier-policy.ts, in a function with no model in it:
// 1. Very low confidence → QUEUE. Hiding uncertain mail is the worst failure.
if (f.confidence < 0.5) return "QUEUE";
// 2. Urgent AND sure → wake the user.
if (f.urgency >= 0.7 && f.confidence >= 0.7) return "PUSH";
// 3. Anonymous, no clock, trivially reversible → SILENT (narrow: marketing only).
if (f.senderTrust < 0.2 && f.urgency < 0.2 && f.reversibility > 0.9) return "SILENT";
// 4. Reversible, very sure, not urgent, trusted → AUTO.
if (f.reversibility >= 0.85 && f.confidence >= 0.85 && f.urgency < 0.5 && f.senderTrust >= 0.5) return "AUTO";
// 5. Default → QUEUE. "I'll look at it on my own schedule" is the dominant bucket.
return "QUEUE";
That's the whole decider. Every threshold is a named constant in one object above it, not a magic number sprinkled through a prompt. Order matters — earlier branches win. I can read this in thirty seconds, write a unit test for each branch, and change the policy without touching the model or re-running an eval.
Try doing any of that to "I asked GPT-4o to pick a tier and it picked QUEUE." You can't test it. You can't diff it. You can't explain to yourself why message #4,012 got hidden. The decision isn't anywhere — it's smeared across a weight matrix and a paragraph of prompt.
Why "consistency over genius" falls right out of it
Once the model's only job is to score four signals, the question stops being "which model reasons best about email policy?" and becomes "which model reads the same four signals the same way every time?"
Those are different questions with different answers. The first one points you at the biggest, most expensive model. The second one points you at a cheap, fast, low-variance one — because a frontier model's extra reasoning, applied to a 30-word email, mostly buys you more ways to have an opinion, which is variance, which is the enemy when you've already moved the judgment into a rule. That's why the cheap model won the last post. It wasn't a cost compromise. Splitting scorer from decider is what made the cheap model the correct choice, not just the affordable one.
And because the contract is "four features → tier" and nothing else, the model isn't load-bearing for correctness — it's load-bearing for perception. Proof: when the LLM is down or rate-limited, a keyword fallback produces the same four features with zero model calls, and the exact same rule runs on top. The plumbing doesn't change. The only thing a better model buys you is sharper feature scores on the genuinely ambiguous mail — which is why the one place I'll spend a frontier model is a dial that escalates only the low-confidence tail, and nowhere else.
What this buys you that "let the model decide" can't
Auditability. The policy is a file. Code review covers it. A regression test pins every branch.
Stable learning. When I correct a misclassification, the correction doesn't go fight the model for control of the answer. It becomes an example that nudges the feature scores toward the right values, and the rule — the spine — stays fixed. The thing that learns and the thing that decides are separated on purpose.
A blast radius you chose. AUTO's thresholds sit deliberately high (reversibility ≥ 0.85, confidence ≥ 0.85, trusted sender) so the system structurally cannot auto-handle a destructive or low-trust action. That floor is a number I can point at, not a behavior I'm hoping the model keeps exhibiting.
The honest part
This doesn't make the model's judgment good — it makes the decision layer honest. Garbage feature scores still produce garbage tiers; the rule only guarantees that identical scores always map to the identical tier, and that I can see why. The thresholds were hand-tuned against 50 emails, and calibrating them from accumulated real corrections is still ahead of me, not behind. The keyword fallback, by design, can't emit PUSH — so a total LLM outage degrades urgent mail to "visible in the queue," never "silently hidden," but it does degrade. I'd rather write that down than pretend the split is free.
The takeaway is portable
This isn't really about email. Any time you're handing an LLM a decision with consequences, you can ask the same question: does the model need to decide, or does it need to read? Separate "what the model perceives" from "what the system does about it." Put the second half in code you can read, test, and stand behind. You get auditability, you get to use a cheaper model, and you stop being surprised by your own product.
The judge, the rule, and the thresholds are all in the open — AGPLv3: github.com/k08200/klorn. The decider is packages/api/src/tier-policy.ts, about sixty readable lines. Go see how few of them there are.
Top comments (25)
Pulling the decider into a readable file and hardcoding the irreversible floor is the part most people never reach, so this is already past the usual argument. The seam left is the one nazar pointed at, and it is sharper than a poke. Three of your four scores describe the email: senderTrust grounded on a recency window, reversibility about blast radius, urgency about the clock. Those have a source outside the model's opinion of itself. confidence does not. It is the model grading its own work, and branch 4 lets it gate AUTO at 0.85. A polished impersonation the model is sure about is exactly a high-confidence, high-senderTrust, reversible-looking email, and it walks into AUTO through the one feature the model authors about itself. The decider is honest wherever the feature's source is independent of the action's blast radius, and scenery wherever it reads a number the model assigned to its own certainty. AUTO wants a corroborator the model cannot write, not a confidence it can.
@jugeni "a corroborator the model cannot write, not a confidence it can" is the cleanest framing of this anyone's given. You're right that confidence is the one feature with no source outside the model's opinion of itself, and branch 4 leans on it at 0.85. What bounds it today: AUTO is classify-only and only ever maps to reversible/internal actions — the irreversible three are floored regardless of any score. So a confident impersonation that reaches AUTO gets quietly handled in a recoverable way, never anything unrecoverable. The fix is your framing: corroboration the model can't author (sender-history consistency, action-type reversibility by lookup) instead of self-graded confidence. That's the next post. (And I owe an adversarial eval case for exactly it.)
The auto-only-maps-to-reversible floor is the move that actually does the work here, and I want to underline it because it gets lost when people talk about confidence thresholds. Your branch 4 sits at 0.85 not because you trust the number, but because the worst case behind that number is bounded by category, not by score. That is the part most agent-guardrails posts skip.
On the corroborator: the cleanest split I have seen is treating it as a separate input channel the classifier reads but cannot write to. Sender-history consistency and action-type reversibility lookup both belong to the runtime, not to the model. The classifier proposes; the runtime arbitrates with facts the model has no access to author. Pulling those into a named external-context object in your trace makes the bound provable after the fact, which is also where adversarial eval gets its teeth.
Looking forward to the follow-up. If you do one adversarial case, the one I would pay attention to is a sender impersonation that lands in AUTO with 0.92 plus an action that the runtime reversibility table marks internal-only. That is the exact corner where the floor is doing all the work and the score is doing none.
@jugeni "bounded by category, not by score" is exactly why 0.85 was never the load-bearing part — the reversible-only mapping does the work, the threshold just rides on it. And "a separate input channel the classifier reads but can't write to" is the cleanest version of the whole thing: sender-history consistency and action-type reversibility live in the runtime, the model proposes, the runtime arbitrates with facts it can't author. Naming that as an external-context object in the trace is the detail I'd have skipped — it's what makes the bound provable after the fact instead of asserted. Going straight into the design.
And the case you'd watch — impersonation landing AUTO at 0.92 against an action the reversibility table marks internal-only — is the exact fixture I'll build the eval around. Score does nothing, floor does everything: the thesis in one test case. (Caught the queue-not-log note too — glad it reads clean. Thanks for how much you moved this.)
"Score does nothing, floor does everything" is a cleaner way to say it than I did. Looking forward to seeing how the impersonation fixture behaves in practice.
@jugeni the meta-canary is the right shape and it slots into something that already exists — there's a daily decision-metrics drift tripwire running off the decision ledger. Confidence-vs-gate disagreement rate on cooperative traffic is exactly what it's built to watch, so once the canary queue lands (#678) the meta-canary is a tripwire on that queue's growth, not a new system. Adding it to the issue. "Same pattern one floor up" — and the floor below already has the plumbing.
The plumbing-below-plumbing observation is the one I'd push on. Meta-canary tripwiring on queue growth assumes the queue itself is live. Silent queue reads as "no drift to flag," but it also reads as "tripwire stopped writing to the queue." Same absence, opposite implications.
The repair we ended up with is a heartbeat on the base tripwire: a synthetic drift sample injected daily, expected to trip. If the canary of the canary stops firing, the base is dead, not quiet. Costs one false positive a day for peace of mind about the silence.
"Floor below has the plumbing" holds. Adding: the plumbing needs a pulse the meta layer can read.
Went and checked against our own tripwire after reading this, and you're right on the nose — it only sees organic traffic, so a dead feed and a quiet feed look identical to it right now. "Canary of the canary" is a good name for it too, stealing that. Filed it: github.com/k08200/klorn/issues/742 — daily synthetic-drift injection, alert if the expected trip doesn't fire. Thanks for the plumbing-below-plumbing framing, it's the right level to think about this at.
Filing that as a github issue is the operational move I would want too, and I would push the drift profile design in two directions instead of one: high-signal (guaranteed trip) and near-threshold (calibration edge).
Guaranteed-trip catches whether the tripwire fires at all, which is the binary question your issue asks. Near-threshold catches something quieter: whether the tripwire fires where it used to. A tripwire whose calibration silently walks from 60% to 40% still passes the guaranteed-trip test every day while progressively losing edge cases in production. The gap between the two profiles is the tripwire's own health signal.
Cheapest form I have found is a small rotating fixture of samples pinned to known-good and known-bad boundaries, and an alarm when the same fixture flips verdict between weeks.
You're right, and I like the framing better than what we have. Every tripwire we've got right now — judge-health.ts's fallback-rate alarm, the per-tier eval-floors gate, the calibration-snapshot drift numbers — is guaranteed-trip: some aggregate crosses a line, alarm fires. None of them would notice a floor that's still being cleared but by less margin every week.
The rotating fixture is the right cheap version of this. Filing it as an issue — will link back here once it's up.
The rotating fixture catches the case that would otherwise never trip. It still won't catch margin erosion on its own unless the fixture logs the distance from threshold, not just pass or fail. A floor that's cleared by 40% one month and 4% the next reads identical to a binary check until the day it flips. Track the margin as a series alongside the fixture and the drift becomes visible weeks before the alarm would.
this is the cleanest statement of the split i've seen, and the part i'd underline is that the decider being a readable file is what lets you use the cheap model, not despite it. once the model only scores, "which model reasons best about policy" becomes "which model reads the same signals the same way," and those point at different models. you said it but it's worth saying twice because almost everyone reaches for the big model exactly where variance hurts most.
the seam i'd poke at is the one your honest section already opens: the rule made the decision auditable, but the four scores feeding it are still a single model's say-so with no independent check. you moved trust out of the decider and it landed in the scorer. that's strictly better, the blast radius is bounded by the AUTO thresholds, but "identical scores map to identical tiers" is only as good as the scores, and those are the unaudited input now.
the one axis i'd treat differently is reversibility. the other three (senderTrust, urgency, confidence) are genuinely perceptions of the email, fair to ask the model. but reversibility isn't a property of the message, it's a property of the action the system would take if it auto-handles it. that's knowable from the action type without asking the model at all, and asking the model invites it to be talked into "this is reversible" by the email's own framing. a calendar-invite auto-accept is reversible because accepting is reversible, not because the email said so. so i'd be tempted to pull reversibility out of the LLM's four and make it a lookup on the action the tier would trigger, which also hardens your AUTO floor: the thing gating destructive auto-actions stops being model-scored at all. curious whether you considered sourcing reversibility from the action rather than the read.
@txdesk you're pointing at the exact seam, and for the part that actually bites it's already where you'd want it. The gate on irreversible actions isn't model-scored — send_email/permanent_delete/forward_external are a hardcoded floor (the third post), so a model talked into "this is reversible" still can't get past it; the receipt check fails closed regardless of the score. Reversibility-from-the-action is already true at the layer that can hurt you.
Where you're dead right: the AUTO tier gate still reads the model's reversibility score (>= 0.85), and that's an unaudited input exactly as you say. Sourcing it from the action the tier would trigger — not the read — is the clean version, and it's going on the list. The calendar-invite framing is the perfect way to put it: accepting is reversible, the email saying so is not evidence.
that distinction is the part i'd actually underline now: the irreversible floor not reading the score at all is the right design, because that's the layer where a wrong call is unrecoverable, and you took the model out of it entirely. fail-closed on send/delete/forward regardless of the score is exactly where you want zero model discretion.
so the residual is narrower than i first put it: it's only the AUTO gate, where reversibility-score is a convenience input deciding "handle this quietly" rather than a safety input guarding something destructive. lower stakes, but still the spot where the calendar-invite case bites, the model can be talked into auto-handling something it shouldn't, just not into anything unrecoverable. sourcing that last score from the action closes the gap cleanly. good thread, you'd already built the part that mattered most.
@txdesk exactly — convenience input, not a safety input, and the calendar-invite case is the right boundary for it. Action-sourcing that last score is on the list now. Thanks for the sharpening; "zero model discretion at the unrecoverable layer" is the line I'll steal.
that line's yours now, take it. and that's the right place to land: zero discretion where it's unrecoverable, model input only where the worst case is a convenience miss. good thread, this one actually moved my thinking on where to draw that boundary. catch you on the next.
Thank you,I really love that you are offering your knowledg
Splitting the scorer from the decider is the move that makes the whole thing testable, and the rule file really is readable in the thirty seconds you promised. The one piece I keep poking at is confidence, since it's a score the same model hands you about its own other three scores. If the model is quietly overconfident on some class of mail, say a polished phishing attempt that reads as a trusted sender, then the confidence < 0.5 → QUEUE safety net is the one branch that never fires when you'd most want it to. Have you seen the low-confidence tail actually catch the cases you'd worry about, or does it mostly flag the obviously ambiguous ones?
You've found the real edge, and it's the right one to poke at. confidence < 0.5 → QUEUE is a net for known uncertainty — it can't fire on confident-wrong, so a polished impersonation that the model is sure about will sail straight past it. Agreed.
The lever that's actually supposed to catch that case isn't confidence, it's senderTrust — and the prompt deliberately tells the model to ground it on observed mailbox history, which outranks how trustworthy the email reads. A stranger that reads as your CFO still scores low senderTrust when there's history to contradict the surface. The honest gap: first-contact impersonation, where there's no history to ground against, is exactly where that has nothing to stand on — and you're right that confidence won't save you there either.
So the backstop isn't a better score, it's the floor from the third post: a wrong tier still can't trigger anything irreversible without an approved receipt. The blast radius of "confidently misclassified" is bounded to it showed up in the wrong lane, not it sent/deleted something.
Straight answer to your actual question: on my 50, the low-confidence tail mostly flags the obviously ambiguous ones. I have not stress-tested it against deliberate overconfidence, and I wouldn't claim it does that work — that's a measurement I owe, not one I've done.
Pulling the decision out of the model and leaving it to score four signals is the part most people skip. The reversibility score is the smart move here, since it lets you gate automation on blast radius instead of raw confidence, which is exactly where most auto-handlers go wrong. I'd be curious how you calibrate senderTrust over time without it drifting as senders change behavior.
@kartik-nvjk that drift is the failure I was most worried about, so senderTrust isn't a running average — it's grounded on a recency window (only the sender's most recent N messages form a prior), override priors carry a TTL and expire ("tastes drift" is the literal comment), and SILENT is excluded from the prior shortcut so a stale prior can never silently hide mail. So it re-grounds as behavior changes instead of calcifying. The open part is choosing the window/TTL from data rather than by hand — same calibration-governance question a couple of others raised.
The scorer-vs-decider split is the same pattern I landed on for an LLM-based routing layer in a production agent. The model outputs a confidence and an intent_signal, the actual routing table is deterministic code that nobody touches during model upgrades. What you describe with the keyword fallback producing the same four features is the clearest proof that the model was never the load-bearing component - it is a perception layer on top of a decision layer, and the two being separable is what makes the system testable. Your reversibility signal is something I have not seen named explicitly before but it is exactly the right axis for anything that touches irreversible state.
the convergence is the reassuring part — independent teams landing on "model scores signals, a deterministic routing table decides, and nobody touches it during model upgrades" is a better signal than any benchmark. You also put your finger on exactly where this goes next: reversibility isn't just a tiering input. The three actions that touch genuinely irreversible state — send, hard-delete, forward-external — don't ride on the classifier at all; they sit behind a deterministic floor that pins the approved bytes and verifies at execute time. Confidence is enough to decide a tier; it's not enough to do something you can't take back.
Ended up writing it up as the third post: dev.to/k08200/confidence-is-enough... — your comment is quoted at the top of it. Thanks for the push.
wait What? you trust it in the first place? it's a large lying machine
Ha — that's the whole title. I don't. It scores four numbers and a deterministic rule makes the call; the "large lying machine" never picks the tier and can't touch anything irreversible. Trust the rule you can read, not the model.