TL;DR: most production AI tasks are not LLM tasks. To triage my email, I replaced a 7-billion-parameter model with a tiny classifier in Go. The rule fits in one sentence. Rules first, a small model next, the LLM only as a last resort. The result: no GPU, sub-millisecond inference, and a cloud call that became rare. Here is how, with the real numbers.
This article is for developers who put an LLM in production and pay the bill. Not a demo.
Most AI tasks are not LLM tasks
In 2026, the default reflex is to wire a big model into everything. A question comes in, you call the LLM. But many tasks do not need it. Filing an email under "work" or "newsletter" is classification. A problem solved for twenty years, long before LLMs.
To classify is to pick a label from a short, stable list. To generate text is something else. The first job needs a small model. The second earns a big one. The rule I defend fits in one sentence. Put the LLM last.
The setup
I built an agent that triages my inbox. It is a daemon. It reads new messages and files each one into a category: work, notification, newsletter, promo, and a few more. Nothing secret, just my real mailbox, with years of mail.
The first version handed every email to a local LLM. A 7-billion-parameter model, Qwen 2.5 7B, served by Ollama on a GPU. Ollama is a tool that runs an LLM on your own machine. It worked. But the price was heavy. A GPU on all the time. One more container to watch. And an absurd slowness for the question asked.
One day I asked myself: does deciding "is this a newsletter?" really need 7 billion parameters? No. The answer sits in two or three words from the sender and the subject. So I rethought the whole thing.
Three layers, from cheapest to most expensive
Every email goes through three layers, in order. It stops at the first one that can answer.
- Deterministic rules. Instant, exact, no cost.
- A small model. Sub-millisecond, on CPU.
- The LLM. Only if the small model is unsure.
The routing code fits in a few lines. It tells the whole story.
func (c *Classifier) decide(m Message) Decision {
// 1. Deterministic rules: obvious senders, decided at the door.
if d := preClassify(m); d != nil {
return *d
}
// 2. Small ML model: sub-millisecond, on CPU.
pred := c.ml.Predict(m)
if pred.Confidence >= c.threshold {
return decisionFrom(pred)
}
// 3. LLM last: only the uncertain tail reaches the cloud.
return c.classifyWithLLM(m)
}
The logic is simple. Each layer costs more than the one before. So each layer only handles what the others could not decide.
Rules skim off the easy mail
A large share of my mail is obvious from the sender alone. A known newsletter address is always a newsletter. A platform alert is always a notification. No intelligence required.
A short list of rules decides these at once. It looks at the sender domain and sets the label. Zero guessing, zero model call, zero cost. These emails never reach the small model, let alone the LLM.
Why not let the model do it? Because a rule you can read beats a prediction you cannot, when the answer is obvious. A rule is stable, testable, and free. You keep it for everything that is certain.
A small model for the ambiguous middle
For the rest, the part that is not obvious, I use two old techniques. TF-IDF and logistic regression.
TF-IDF turns text into numbers. Each word gets a weight based on how common or rare it is. Logistic regression is a simple model. It learns to separate categories from those numbers. Together they make a solid, light text classifier.
I train it in Python, with scikit-learn, on nearly 5,800 labelled emails across 6 categories. Then I export it to a plain JSON file. That file weighs 2.4 MB. Compare it to the 7B model: several gigabytes and a GPU.
The key point: inference is 100% Go. No Python, no GPU, no C dependency. The Go code reads the JSON and predicts in well under a millisecond, on a plain CPU. It all fits inside my production image, a distroless image with no shell and no system tools.
And the accuracy? In 5-fold cross-validation, the model reaches 81% correct. Cross-validation splits the data into 5 parts. You train on 4 and test on the 5th, in turn. It is an honest measure, taken on mail never seen during training. The model is strong on the frequent, clear categories, around 0.88. It is weaker on the rare, fuzzy ones, around 0.62.
Finally, the model returns a confidence, between 0 and 1. Above 0.60, I trust it. Below, the email moves to the next layer.
81%, so what?
On its own, 81% looks mediocre. It is not a problem, because of the cascade. No layer has to be perfect. Each layer just has to do what it is good at.
The rules decide the easy mail, without error. The small model handles the bulk of the middle, with confidence. The LLM only sees the tail, the truly ambiguous mail. The expensive call becomes rare. That is the whole point.
You are not chasing a perfect model. You are chasing a system where cost follows difficulty. An obvious email costs nothing. A hard email costs one cloud call. And there are few hard emails.
The token parity trap
There is one tricky part. The model trains in Python but runs in Go. The way you cut text into tokens must be identical on both sides. Byte for byte.
A token is a piece of text the model counts, usually a word. If the Go split differs from the Python split, even slightly, the model sees tokens it never learned. Accuracy drops in silence. No error, just worse predictions.
My fix: a hand-rolled tokenizer, duplicated exactly in both languages. Same regular expression, same table to strip accents, no external unicode library. One explicit table, the same in Python and in Go.
// The same table lives in the Python trainer.
// One divergence and the model sees tokens it never learned.
var fold = map[rune]string{'é': "e", 'è': "e", 'ç': "c" /* full table */}
var wordRE = regexp.MustCompile(`[a-z0-9]{2,}`)
func wordTokens(s string) []string {
var b strings.Builder
for _, r := range strings.ToLower(s) {
if rep, ok := fold[r]; ok {
b.WriteString(rep)
} else {
b.WriteRune(r)
}
}
return wordRE.FindAllString(b.String(), -1)
}
A parity test compares the two tokenizers on real text. If they diverge, the build breaks. A red build beats an accuracy that melts without warning.
The silent regression that taught me a guard-rail
The model improves through a simple loop. When the agent files an email in the wrong place, I move it to the right folder. That move becomes a new label. I retrain, and the model learns from my correction.
One day, this loop bit me. I had re-sorted my mail by hand. One category fell below the minimum number of examples needed to learn it. The retrain dropped it in silence. The new model could no longer predict it. Still no error shown.
The fix: the training script now refuses to drop a category I rely on. It aborts and tells me which folder ran dry.
# Retraining refuses to lose a category in silence.
python train.py --in labeled.jsonl --out model.json \
--expect-labels "work,notification,newsletter,promo,home"
# ABORT: expected classes would be DROPPED from the model: [personal]
The lesson goes beyond email. A silent regression is worse than a crash. A crash, you see. An accuracy that quietly drops, you find out too late. So you make it loud, on purpose.
When the LLM earns its place
I did not delete the LLM. I moved it to where it earns its cost. Writing.
Classifying an email is a closed problem, with few answers. Writing a human reply is an open one. And free language is exactly what a big model does better than anything. So when the agent has to draft a real message, a cloud model handles it.
Small model for the closed task. Big model for the open task. The right tool at each stage. That is the real lesson, not "LLMs are bad". LLMs are excellent. Just not for everything.
The checklist before you reach for an LLM
Before you call a big model on reflex, run the task through these questions.
- [ ] Does the task have a small set of stable answers? Then it is classification, not an LLM
- [ ] Can you write rules for the obvious cases? Do them first, they are free
- [ ] Do you have labelled examples? A small model is probably enough
- [ ] Does the model need to understand free language, or just pick a label?
- [ ] Can you measure honest accuracy, with cross-validation?
- [ ] Does the small model run without a GPU, on CPU, inside your production image?
- [ ] Is your tokenizer identical at training and inference, byte for byte?
- [ ] Does a guard-rail prevent a silent regression on retrain?
- [ ] Do you keep the LLM for what it truly does better, generating language?
What to remember
The 2026 reflex is the big model for everything. Often, the task does not need it. A cascade costs far less and runs far faster. Rules for the obvious. A small model for the middle. The LLM for the only real difficulty.
This is not a rejection of AI. It is engineering. You match cost to difficulty, layer by layer. Building an AI system and watching the bill climb? Want to know where a small model would replace your LLM? That is exactly what I do. Write to me. Keep the big model for what deserves it.
Originally published at jrobineau.com.
I'm Jules Robineau, a senior Go backend and DevSecOps freelancer based in Paris. I build and harden production AI/backend systems at scale (25M+ users). CompTIA PenTest+, Top 1% TryHackMe. Services · GitHub · LinkedIn
Sources: scikit-learn, TfidfVectorizer, scikit-learn, LogisticRegression, scikit-learn, cross-validation
Top comments (5)
The 81% number is the most useful thing in this post and I suspect it'll be the most misread. On its own it looks weak. Paired with a confidence threshold and an escalation path, it's not an accuracy figure at all — it's a coverage figure. The question isn't "how often is the small model right," it's "how much traffic can it absorb at a confidence level where being wrong is cheap."
We landed somewhere similar routing scraped items into buckets before anything expensive touched them. The surprise for us wasn't the cost drop, it was latency variance: the deterministic layers made p99 predictable in a way no amount of prompt tuning did, because most requests stopped answering to a network call at all.
One thing I'd push on: TF-IDF plus logistic regression degrades quietly as your category distribution shifts. Do you have anything watching the escalation rate over time? A creeping "% sent to the LLM" is usually the first sign the small model has gone stale, and it shows up long before accuracy does.
Coverage, not accuracy: exactly, and it only means something next to the 0.6 threshold and the fallback (below it the email goes to the cloud model, so an unconfident miss is cheap). Same on latency for me: the deterministic rules up front short-circuit most messages before any network call, which is what made p99 predictable.
And you caught the real gap: I wasn't watching the escalation rate over time, something I'd flagged but hadn't built yet. You're right it's the leading indicator, the model goes unconfident before it goes wrong. So I wired it: every ML decision now logs {ts, confidence, escalated}, aggregated by ISO week, with an alert when the recent rate climbs past baseline. I also added a retrain gate that blocks a model whose macro-F1 dropped vs the deployed one, since my old guard only caught a class disappearing, not one whose F1 quietly collapses.
Thanks for the push, that watch is the piece I should've had from day one.
The 0.60 threshold is doing a lot of load-bearing work in this design, and it's worth checking whether it means what you think it means.
A confidence cut only routes correctly if the number is calibrated — if predictions at 0.60 are actually right about 60% of the time. Logistic regression is better behaved here than most alternatives, since it's fit on log-odds directly rather than having probabilities bolted on afterwards like an SVM or a forest. But two things in your setup push against that: six classes with very uneven support, and a retrain loop fed by your own corrections, which shifts the label distribution every cycle. Both move the intercepts and can quietly decalibrate the tail.
Cheap way to check: bin your held-out predictions by confidence — 0.5–0.6, 0.6–0.7, and so on — then compare each bin's actual accuracy against its mean confidence. If the 0.6 bin comes in at 0.45, the threshold is waving through errors it should be escalating, and the aggregate 81% will never show you that.
Which I think also answers James's question above about the tail: set the threshold per class rather than globally. Your clean categories sit near 0.88 and the fuzzy ones near 0.62, so one global cut is simultaneously too strict for the easy classes and too loose for precisely the ones where a mistake is expensive. Pick each class's cut to hit a target precision on held-out data, and let the escalation rate fall out of that rather than choosing it up front.
sklearn.calibration.calibration_curvegets you the plot in about five lines. I'd look at the curve before reaching forCalibratedClassifierCVthough — on logistic regression it may already be close enough that per-class thresholds are the whole fix.The tokenizer parity test that breaks the build on divergence between Python training and Go inference is the most production-hardened detail in this post — cross-language tokenization drift is exactly the kind of bug that shows up after a dependency update on the Python side, not at training time. The same discipline applies whenever you split train/serve across runtimes: the contract isn't the model file, it's the tokenizer, and enforcing that in CI is the right place for it. The feedback loop design where moving a misclassified email becomes a labeled training example is elegant because it aligns the human correction action with the data pipeline input. The only extension I'd add is time-weighting the sample to prevent older mail patterns from diluting recent distribution shifts as the label set evolves.
"Most production AI tasks are not LLM tasks" is the sentence I wish more teams internalized before reaching for the biggest model by reflex. The sub-millisecond, no-GPU result is the obvious win, but the quieter one is debuggability: a rule and a small classifier fail in ways you can actually inspect and unit-test, versus a 7B model where "why did it file this as newsletter" is unanswerable. The one place I've seen this architecture strain is the tail — the labels that are genuinely ambiguous even to a human. How do you decide what falls through to the LLM as the last resort versus just getting a low-confidence label from the small model?