I let GPT-4o and a cheaper model fight over my inbox. GPT-4o lost.

Here's the scoreboard. Same 50 emails, same prompt, same 4-tier task:

Model	Accuracy	Note
google/gemini-2.5-flash	88%	100% recall on urgent mail — never missed one
google/gemini-2.5-pro	82%	the "smarter" sibling
openai/gpt-4o	82%	the reflex pick
anything cheaper than flash	< 80%	failed my floor, didn't ship

The cheap model didn't tie the expensive ones. It beat them by six points and never missed an email that should have woken me up. The two models I'd have reached for on instinct — the obviously-smarter ones, the ones that cost several times more per token — both came second.

I almost didn't run this comparison at all. That's the part worth your time.

The task

I'm building an email firewall. Every inbound message gets exactly one of four tiers:

• SILENT — recorded, never shown (marketing, receipts, FYI)
• QUEUE — visible when I choose to look, no notification
• PUSH — actually interrupt me
• AUTO — reversible, hands-off (classified only, for now)

That's the entire output surface. No suggestion cards, no "AI thinks you should reply" badges. One label per email.

The thing doing the labeling is what I call the judge. It's the part I'd assumed needed a good model — reading an email and deciding whether it's allowed to ring your phone feels like a judgment call, and judgment calls are what you buy a frontier model for.

So I had gpt-4o wired in and I was ready to leave it there. Then I did the boring thing and measured it.

What "measured" means here

The eval set is committed to the repo: packages/api/eval/judge-eval-set.json. Fifty emails, synthetic and PII-free, hand-labeled to encode one specific person's policy — QUEUE is the default, SILENT is narrow (clear marketing only), PUSH is urgent and confident, AUTO is reversible and not urgent. The tier mix is 21 QUEUE / 13 PUSH / 12 SILENT / 4 AUTO.

One command runs a model against it:

pnpm eval:judge   # tsx scripts/poc-accuracy.ts --in=eval/judge-eval-set.json

I ran it across the models I was actually choosing between. Flash won. Not "won on cost, tied on quality" — won on quality, and it happens to be the cheap one. I pinned production to it and wrote the result into the commit message so future-me can't quietly pretend the expensive model was a sacrifice I made for the budget:

flash is not a compromise — it scores 88% / 100% PUSH recall, beating gemini-2.5-pro and gpt-4o (both 82%).

Why cheap wins here

A frontier model sells you reasoning depth. Long chains, hard problems, hold-ten-things-in-your-head problems. Email triage is none of that. It's a short, repetitive, read-four-signals-and-be-consistent problem. You're not paying for capability that moves this needle — you're paying for capability you never touch, and a bigger model's extra "thinking" mostly buys you more chances to overthink a 30-word email.

There's an architecture reason too, and it's the load-bearing one. The LLM never picks the tier. It scores four features per email — confidence, sender trust, reversibility, urgency — and a ~20-line deterministic rule maps those four numbers to PUSH/QUEUE/SILENT/AUTO. The model is a feature-scorer, not a decider. So I don't need a model that reasons brilliantly about email policy. I need one that reads four signals the same way every time. Consistency, not genius. That's exactly the job a cheap fast model is good at — and exactly the job where a bigger model's cleverness becomes variance you don't want.

It also means the policy is auditable without the model in the loop. I can read the rule. I can test it. If the model's down, a keyword fallback produces the same four features so urgent mail still gets through. None of that works if you let the LLM free-hand the answer.

Before you @ me

This is 50 emails. It's small on purpose — it's one person's mental model written down, not a benchmark, and I'm not going to dress it up as one. The set is synthetic, so it tests whether the model applies my policy consistently, not whether it can read the real world. A different inbox with a different owner would draw the lines somewhere else and might rank the models differently.

I'm also only claiming what I measured. Flash hit 100% recall on PUSH — it never sent an urgent email to a quiet tier. I'm not going to invent per-tier numbers for the models that lost; I have their headline accuracy and that's what I'm putting my name on.

What I'm not walking back: on the one task I actually care about, on the set that's sitting in the public repo for you to open, the expensive models lost. That result was stable enough to bet production on.

The lesson is annoyingly cheap

Most of us never run this comparison. "Use the best model" is the default, the best model is the expensive one, and the leaderboard agrees, so why would you waste an afternoon proving the obvious?

Because the leaderboard has never seen your task. MMLU doesn't know what you mean by "urgent." The only eval that ranks models on your problem is the one you write — and when you write it, the ranking stops matching the price tag surprisingly often. The frontier model isn't smarter at your job. It's just smarter at the jobs in the press release.

Write the small eval. Run the cheap model against it before you reach for the expensive one. Worst case you confirm the obvious. Best case you cut your bill and your accuracy goes up, which is a sentence I didn't expect to type either.

The judge, the eval set, and the deterministic rule are all in the open — AGPLv3, OpenAI-compatible, point it at Ollama or vLLM and keep your mail on your own box: github.com/k08200/klorn. The eval set is packages/api/eval/judge-eval-set.json. Open it, label it your way, and go find out which model actually wins on your inbox.

Comments

[Nazar Boyko]

Flash winning is the fun headline, but the sentence doing the real work is the one where the model only scores four features and a tiny rule picks the tier. Once the LLM is a feature scorer and not the decider, "consistency over genius" falls right out of it, and a cheap fast model is exactly what you want for reading the same four signals the same way every time. That framing deserves its own post more than the price comparison does.

[yongrean]

@nazar_boyko nailed it. The price headline is the hook; the feature-scorer-not-decider split is the actual thesis. Once the model only reads the four signals and a deterministic rule does the deciding, the policy is auditable and testable without the model in the loop — and a cheap fast model stops being a cost compromise and becomes the correct tool for reading the same signals the same way every time. Cheapness is a side effect, not the point. You basically wrote the intro to the next post. Writing it now.

[Mudassir Khan]

the PUSH recall number is the one i'd want above the accuracy headline. 82% that includes a missed PUSH fails the firewall test. 88% with 100% recall doesn't.

we built something similar for a support routing tool — four signal read to a deterministic bucket rule. the win: you can QA the rule independently of the model. found a logic error in the rule (wrong threshold on the 'plan tier' signal) that the model's accuracy was masking. 20 minutes with a unit test.

does the deterministic rule get retrained from the eval data, or is it hand tuned when your own policy shifts?

[yongrean]

@mudassirworks 100% on the recall point — that's why it's not just a reported number, it's a hard safety invariant in the eval: a missed PUSH must degrade to QUEUE (visible), never SILENT (hidden). The aggregate can wobble; "an urgent mail went silent" is a test failure, full stop.

And your plan-tier bug is the best argument for the split I've heard — the model's accuracy masking a rule error is exactly the failure an end-to-end number hides and a unit test on the rule catches. That's the whole reason the decision lives in code and not in the weights.

To your question: it's hand-tuned, not retrained. The thresholds are named constants fit once to my 50-email ground truth, and I change them deliberately when my own policy shifts — the eval set is a regression gate on the rule, not training data for it. There's an approved-override merge layer, and the constants are written to be the surface a future calibration pass (from accumulated real overrides) can tune — but I'm keeping that a reviewed step, not online fitting. The moment the rule auto-fits to data, you lose the exact property that let you catch your plan-tier bug in 20 minutes: that a human can read it and pin it with a test.