I spent a weekend running two small language models against three jobs I
would normally hand to a language model without thinking about it. For one
of those jobs I also wrote a regular expression, as a control.
The regex won. Thirty test cases out of thirty, in zero milliseconds, from
a pattern that took an afternoon to write. The better of the two models
scored twenty-eight, took five seconds a call, and the two it missed are
both mistakes a regex is structurally incapable of making.
That is the kind of result I was looking for, and the question behind it is
one I keep avoiding: what is the smallest amount of computation that
reliably does this job? Not the cheapest model. Not the best small model.
The smallest thing that works, which means the ladder has four rungs and the
bottom one is not a model at all.
- Deterministic code. A regex, a lookup table, an if statement.
- A small local model. Runs on hardware you already own.
- A larger local model. Needs a GPU.
- A remote frontier model. Someone else's hardware, metered.
Most projects start at rung four and never ask about the others. This is a
weekend spent asking.
The hardware was a Raspberry Pi 4, 4 GB of RAM, heat sinks and a fan, a
hundred and seventy dollars most of which I already owned. The models were
smollm2:360m and qwen2.5:1.5b through Ollama. The test cases are
ninety-five questions about a fictional small winery, so nothing can be
looked up and every answer can be judged by reading. Sixteen of the fifty
classification messages are deliberately ambiguous, because real inboxes are
mostly ambiguous.
Four verdicts, assigned per workflow, to force a result you can act on
instead of a number:
- Unnecessary: deterministic code already does this.
- Sufficient: a small local model does it well enough to run unattended.
- Marginal: usable with a person checking the output.
- Insufficient: the error profile or the latency is not acceptable.
And one rule, which is the part I would insist on if you copy nothing else:
I wrote predictions for every workflow and committed them before I
installed Ollama. Twenty-two of them, each marked confident, leaning, or
coin flip. Once there is a timestamped expectation there is a defined moment
when you are done, and no room to decide afterward that you knew all along.
I will come back to how they scored, because it is not flattering.
Extraction: pulling an order number out of a message
Find ORD- followed by six digits, or report that there isn't one. Thirty
cases: twenty-two with an identifier, five without, and three decoys. The
decoys are a membership number, a UPS tracking number and a phone number,
all of which look like identifiers and none of which are order numbers.
| regex | qwen2.5:1.5b | smollm2:360m | |
|---|---|---|---|
| Overall | 30/30 | 28/30 | 8/30 |
| Declined correctly when absent (8) | 8 | 8 | 0 |
| Decoys ignored (3) | 3 | 3 | 0 |
| Invented an answer | 0 | 0 | 7 |
The regex won. Thirty out of thirty, in zero milliseconds, at zero watts
above idle.
That result is boring until you read the two the best model missed.
On ORD-999000: wrong address, please hold. it answered 999000, dropping
the prefix. Annoying, recoverable, the kind of thing you fix with a
post-processing step you should not need.
On ORD-111222 please refund it answered NONE.
That is the shortest and least decorated input in the whole set. Almost the
entire message is the order number. And the model returned a confident,
correctly formatted assertion that there is no order number present. In a
pipeline, that is a refund request for an order that exists arriving at a
system that has decided it does not.
A regex cannot make that mistake. Not "rarely does" — cannot. The two
errors separating 28 from 30 are both of a class that the deterministic
baseline is structurally incapable of producing, which is a different and
much stronger statement than a two-point accuracy gap.
The 360M result needs two numbers, not one
By the contract, smollm2:360m scored 8 of 22 on the cases that contained
an identifier. That reads like a model that cannot find things.
It is not. All fourteen of its wrong answers are the correct six digits
with ORD- stripped. ORD-556120 came back as 556120, fourteen times
out of fourteen.
It located the right identifier in 22 of 22 and formatted it correctly in
- Those are two different skills and the single score hides one of them. The model understood the task perfectly and could not obey the output contract, which is a completely different engineering problem than "the model is too small," and it has a completely different fix. (Constrained decoding. I have not tested it. It is the obvious follow-up and it is not in this post.)
The decoys follow from the same gap. It handed back MEM-443322,
1Z999AA10123456784 and 503-555-0142 verbatim, all three. It has no
model of the format at all. It returns whatever in the message looks most
like an identifier.
Which suggests a use for it that is not extraction
A model with perfect recall and no precision is not a bad extractor. It is
a candidate generator that has been asked to do the wrong job.
Twenty-two of twenty-two located, eight of twenty-two formatted. Put that
output through the regex as a validator rather than trusting it directly,
and you get the model's recall with the pattern's precision. The model
proposes anything that looks like an identifier, including format variants
the pattern was never written for. The pattern rejects anything malformed.
Crucially, neither half of that pipeline can produce the error the 1.5B
model produced alone. The model cannot cause a false negative because it
does not decide, and the regex cannot invent because it only matches.
I have not tested that composition and it is not in these numbers. But it
is the one place in the extraction result where a small model has a job
worth giving it, and it is a job where being sloppy is free.
Verdict for extraction itself: Unnecessary. Write the regex.
Classification: routing a message to one of six queues
Appointment, billing, availability, complaint, question, escalate. Fifty
messages. The instruction said to use escalate when the message needs a
person, including when you cannot tell what it is about.
| qwen2.5:1.5b | smollm2:360m | |
|---|---|---|
| Overall | 23/50 (46%) | 15/50 (30%) |
| Labels used | 6 of 6 | 3 of 6 |
| Most common label | complaint, 38% | escalate, 86% |
| Wrong and not an escalation | 26 | 3 (meaningless) |
The 360M model answered escalate to 43 of 50 messages, which is not
classification, and there is nothing further to say about it here. The rest
of this section is the 1.5B model.
It is classifying, and it is not good at it. Forty-six percent, with
twenty-six messages routed confidently to a wrong queue that was not the
safe one.
Read those twenty-six and the verdict writes itself:
- Someone slipped on the steps by the barrel room and hurt their wrist → complaint
- A formal notice regarding a trademark → complaint
- "I have a severe sulfite allergy, what is actually in the wine" → complaint
- A reporter asking about the 2024 harvest → question
- A distributor wanting to carry the wine in forty stores → question
An injury report, a legal notice and an allergy question, all routed to the
complaints queue, confidently, with no escalation. All five billing
messages about money problems also went to complaints.
Forty-six percent sounds like something you could work with. Twenty-six
confident misroutes including an injury and a legal notice is not something
you can work with.
And then I broke the score down by class
This is the part I nearly missed, and it is the most useful thing in the
experiment.
| Label | Correct | Of |
|---|---|---|
| complaint | 6 | 6 |
| question | 9 | 10 |
| availability | 3 | 6 |
| appointment | 4 | 9 |
| billing | 1 | 7 |
| escalate | 0 | 12 |
Two classes near perfect. Four poor. One at zero.
The split is not difficulty, and it is not class size. It is what kind of
label each one is.
complaint and question are topics. You can tell what a message is
from the words in it. Someone complaining sounds like someone complaining.
escalate is a routing decision. Nothing in the content of a message
makes it an escalation. It is an escalation because of what should happen
next. "I have a severe sulfite allergy, what is actually in the wine" and
"what time do you close" are both questions about the business, with the
same grammar and the same tone, and one of them is a liability event.
The model did topics at 90 to 100 percent and routing decisions at zero.
And billing sat in the middle for exactly the same reason: a message
about being charged twice is a complaint, as text. The label is about
which department owns the problem, which is not a property of the sentence.
Small models answer "what is this about." They do not answer "what should
we do about it."
That reframes the five examples above. They are not five unlucky
misroutings. All five are escalations, and escalate scored zero of twelve.
The model was never going to find them, because finding them requires
knowing a consequence that the text does not contain.
Two caveats on how hard to lean on this. The classes hold between six and
twelve cases each, so "six of six" is six cases and the individual numbers
are thin. And this is one model. The pattern across all six classes is
consistent enough that I believe the explanation, but I am offering it as
the best account of the data rather than as a measurement.
Verdict: Insufficient. And the actionable version: if one of your
labels means "a human needs to see this," a small model will not find it.
Build that one as a rule.
Grounded QA: answering a question from four documents
Fifteen questions against a 650-token corpus: club policy, tasting room,
production notes, shipping. Eight are answerable from one passage, three
require combining two, and four have no answer in the material at all.
I read all thirty answers by hand against the corpus. That is the only way
to score this, and I will say why in a moment.
| qwen2.5:1.5b | smollm2:360m | |
|---|---|---|
| Single passage (8) | 8/8 | 7/8 |
| Two passage (3) | 1/3 | 0/3 |
| Answerable total (11) | 9/11 | 7/11 |
| Correctly said "not in the material" (4) | 2/4 | 0/4 |
Three boundaries separate cleanly here, and they are the useful result.
Retrieval works
The 1.5B model answered all eight single-passage questions correctly. Six
bottles in the Vineyard tier. Twenty dollars per person. September 9th.
Thirty-four states. No dogs indoors. 21.8 Brix on the 12th. Twenty-five
dollar restocking fee. Ten people on the barrel room tour.
Asked what happens if you join the club and immediately want out, it
returned the two sentences that answer the question and stopped:
Cancellation requires one completed shipment. Members who cancel before
receiving a shipment are charged for that shipment.
That is competent lookup across four documents, on a $170 computer, with
the wifi switched off. I did not expect it and it is the most encouraging
thing in the weekend.
Combination does not
One club member brings five guests on a Saturday. Club policy waives the
tasting fee for the member plus three guests. The tasting room charges
twenty dollars a head. Six people, four waived, two paying: forty dollars.
Both models answered twenty.
Neither could hold "plus three guests" against a party of six and subtract.
The 1.5B model did get the second half right, correctly working out that a
party of six is not a group larger than six and therefore needs no
reservation, which is the only piece of multi-document reasoning either
model performed all weekend.
The second two-passage question is the one I keep thinking about. Asked why
someone would not find a 2024 rose and what they would be offered instead,
the 1.5B model quoted the corpus exactly, and then:
...the reference material states that "We did not bottle a rose in 2024.
The fruit that would normally go to rose was needed to make volume on the
estate Pinot Noir." Therefore, the 2024 rose would be replaced with the
fruit that would typically go into the estate Pinot Noir production.
The quotation is verbatim and right. The next sentence reverses the only
relationship in it. The fruit went into the Pinot Noir, so what you would
be offered is the Pinot Noir. You cannot be offered fruit.
Correct retrieval and inverted inference, in consecutive sentences, with
the evidence on screen between them.
The 360M model's multi-document failures are worse than wrong. They are
contaminated. It took the fourteen-days-notice rule, which governs skipping
a club shipment, and applied it to booking a tasting room reservation. It
reached into the shipping document to answer a cancellation question. Given
more documents to search, it finds more wrong ones.
Knowing the answer is absent is the weakest link
Four questions have no answer in the corpus, chosen to be things a reader
would reasonably expect a winery to have written down: a wine's alcohol
percentage, who the winemaker is, what a club tier costs, what yeast was
used.
The 1.5B model abstained on two and fabricated two. The 360M model
fabricated all four.
And the fabrications are a more dangerous species than invention. Asked for
the alcohol percentage of the 2024 estate Pinot Noir, the 1.5B model
answered:
The 2024 estate Pinot Noir had a Brix range of 23.1 to 24.6 at pick.
Every component of that is true. Right wine, right vintage, right numbers,
right document. Brix is sugar at harvest and has nothing to do with alcohol
in the bottle. It reached for the nearest real fact instead of making one
up, and produced something that reads as expertise while answering a
question nobody asked. If you do not know what Brix is, that answer is
indistinguishable from a correct one.
Both models made the same mistake on what the Cellar tier costs: both
answered "twelve bottles per quarter." Grounding gave them the right tier
and they handed back the quantity in place of the price.
The one answer the evidence did not change
The 360M model, asked whether a dog can come inside the tasting room:
Yes, you can bring a dog inside the tasting room.
The corpus says the patio is dog friendly and the indoor room is not,
service animals excepted. I have the same answer from a run where the model
had no corpus at all, and it is identical byte for byte. Having the right
answer directly in the prompt changed nothing.
In a category where being wrong means a family drives out to a winery and
gets turned around at the door.
Verdict: Marginal. Usable with a person reading the output. Not usable
without one.
What it costs
| Workflow | Model | tok/s | Prompt | Answer | Median call |
|---|---|---|---|---|---|
| extraction | qwen2.5:1.5b | 4.3 | 92 | 9 | 5.0 s |
| classification | qwen2.5:1.5b | 7.5 | 104 | 2 | 2.9 s |
| QA | qwen2.5:1.5b | 3.7 | 708 | 15 | 6.9 s |
Two things in here surprised me.
The compute tax does not track parameter count. The 1.5B model is
slower than the 360M on extraction, 4.3 against 5.9 tokens per second,
and faster on classification, 7.5 against 6.2, because on classification
it answers in two tokens where the smaller model takes six. Four times the
parameters, and which one wins depends entirely on how much each one
decides to say.
A bigger prompt is cheap. A longer answer is what you pay for. I have a
clean comparison here by accident: an earlier QA run went out with 97-token
prompts and the real one with 708. Seven times the prompt added fifteen
percent to the wait before output started, because prompt evaluation
parallelizes. The latency increase came from answers three times longer,
plus a twenty-one percent drop in per-token generation speed from attending
over a longer context. Median latency went from 3.6 to 6.9 seconds and the
slowest single call took ninety-eight.
If you are sizing a local system, that is the inversion to internalize.
Stuffing more context in is affordable. Letting the model be chatty is not.
And check your thermals, which is not optional advice. My first attempt
at this ran with heat sinks and no fan. Thirty calls to a 360M model was
enough to hit the Pi 4's soft temperature limit and start capping the clock.
vcgencmd get_throttled returned 0xe0000. I threw those latency numbers
away.
Nothing in Ollama's output, my scorer, or the result files said a word about
it. Every number on screen was internally consistent and quietly describing
a machine that was slowing down. The only way to find out was to ask the
firmware a question I had to already suspect the answer to. Buy the fan.
And it does run with the network off. I switched the radio off, proved
there was no route out, and ran one call from each workflow. All three
worked. The claim that supports is narrow and worth stating carefully: this
configuration, on this afternoon, ran with no route to the internet. It is
not a claim that local models are private or that this setup is secure, and
a Pi can be misconfigured as thoroughly as anything in a data centre. But
for the specific question of whether the inference needs the network, it
does not.
So: do you actually need an LLM?
| Workload | Verdict |
|---|---|
| Pull a known-format identifier out of text | Unnecessary |
| Route a message to one of six queues | Insufficient |
| Answer a question from your own documents | Marginal |
Three workflows that all sound like language tasks, three different answers,
and not one of them is "yes, use a model."
And the same line runs through all three. Extraction asks what this string
is. Topic labels ask what this message is about. Single-passage lookup asks
what the document says. Those went well, or well enough. Escalate asks what
we should do, the fee question asks what the customer owes, and the
unanswerable questions ask whether we know. Zero of twelve, both models
wrong, and two of four.
A small model will tell you what something is. It will not tell you what
to do about it. Every result here falls on one side or the other of that
sentence, and I did not have it written down when I started.
Which gives four properties worth checking before you reach for one:
- The answer is already present in the input. Not inferable from it. Present in it.
- There is exactly one of it. The moment two facts have to be combined, this stops working, and the arithmetic in that tasting fee question is the proof.
- The output is short. Longer answers are where the latency lives on this hardware, and where the fabrication lives everywhere.
- Something other than the model decides whether the output is acceptable. A regex, a schema, a person, an index that either returns rows or does not.
Take away any one of those four and the experiment says do not do it. The
46 percent and the zero of twelve are what it looks like when you ask a
small model to be the thing that decides.
And if you have all four, you may not need the model either. That is worth
checking first, because it was true for one of my three workflows and I did
not expect it to be.
Where I was wrong
Twenty-two predictions, committed before I installed anything. Ten right,
eleven wrong, one unresolvable.
Two of my seven confident calls broke. I was sure the 360M model would get
most of the easy extraction cases, and it got 8 of 22 by failing at format
rather than at finding. I was sure it would produce invalid output often,
and it did so once in fifty, then emitted the same valid word forty-three
times.
But the number that bothers me is not in the confident column.
| Confidence | Right | Wrong |
|---|---|---|
| Confident | 5 | 2 |
| Leaning | 2 | 7 |
| Coin flip | 3 | 2 |
My hunches did worse than my coin flips. Two of ten on the calls where
I had a feeling but not much behind it, three of five on the ones I
explicitly labeled as guesses.
That ordering is backwards and it is the most useful thing on the sheet. On
the coin flips I knew I was guessing, so I guessed cleanly. On the leaning
calls I was pattern-matching against intuitions about model behavior that
turned out not to describe these models at all, and the feeling of having a
reason was the problem. If I had marked all twelve of those as coin flips I
would have scored better.
I will take that over a clean result. The honest version of this exercise is
not "here are the numbers," it is "here is what I believed, here is what
happened, and here is the gap."
The gap, this time, was mostly in the hunches.
The harness that produced these numbers had five measurement flaws in it,
one of which manufactured an entire finding out of a missing directory and
nearly got published. That is a separate post.
Everything here is reproducible: fixtures, runner, scorer, the committed
predictions and the run record are at
github.com/kenwalger/right-size-the-model.
Fixture set v1. Read LIMITATIONS.md before citing any of these numbers:
two models, one board, one quantization, ninety-five cases. This is an
illustration of a method, not a benchmark.
Top comments (4)
I think there’s another interesting angle here. My philosophy is that we already have decades of algorithms capable of handling diverse, dynamic, and heterogeneous environments deterministically. Constraint solvers, graph algorithms, state machines, optimization techniques, and probabilistic methods are all incredibly powerful when composed correctly.
I don’t think LLMs necessarily need to be the center of the architecture. They can be specialized subprocesses or subkernels, invoked when semantic interpretation, generative reasoning, or ambiguity actually warrants them.
And now with engines like Jev from TypeSafe, things get even more interesting. We can mix and match computational primitives, using deterministic algorithms for orchestration and constraints, probabilistic models for bounded judgments, and LLMs for open-ended reasoning.
The real engineering challenge is figuring out which primitive is best suited for each problem, then composing them intelligently.
Deterministic execution doesn’t require a deterministic world.
I think the bigger question isn’t necessarily whether we need an LLM, but whether we’re architecting systems that use the right computational resources at the right time, with the right level of confidence, cost, and complexity.
Intelligence doesn’t have to be the architecture. It can simply be a resource the architecture knows how to use.
I think you've articulated the architectural extension of what I was trying to explore with this experiment. The question isn't whether LLMs are useful. They clearly are for certain classes of problems. It's whether we've made them the default computational primitive without first asking what the problem actually requires.
"Deterministic execution doesn't require a deterministic world" is an especially useful distinction. Inputs can be unpredictable, incomplete, or ambiguous without requiring every subsequent decision to be probabilistic. We have decades of engineering experience with exactly those conditions, as you point out.
I'd add one dimension to the composition question: authority. It's not just which primitive is best suited to a task, but what the system is permitted to do with its output. A model might interpret an ambiguous request, while a deterministic policy determines whether the resulting action is authorized. That separation matters just as much as choosing the right computational resource.
Jev is incidentally on my list of things to explore. The idea of composing different computational approaches rather than treating the LLM as the center of everything is compelling.
And I particularly like your closing observation. Intelligence can be a resource the architecture uses, rather than the organizing principle around which everything else must be built. That's a useful way to frame the engineering decision.
Your "tell you what something is" line fits a case where we skipped the model entirely. Spotting earnings releases among foreign issuers' 6-K filings is a what-is-this question, and the answer sits in the document itself. So it's rules over the title and the reporting period, plus a count of the metrics that show up. Writing the rules was the cheap part. What made it work was 60 days of real filings cached on disk, so every edit re-ran on all of them in seconds. The first version rejected Nebius's own release, because "will hold an earnings webcast today" read as a pre-announcement. For the routing workflow that came out Insufficient, is there anything a pattern could check, or does a queue label give it nothing to hold on to?
The cached corpus is the most interesting part of your example. Having 60 days of real filings available for every rule change is what lets you discover cases like the Nebius release rather than assuming the pattern is correct because it handles the examples you anticipated.
And yes, two jobs. One is identifying what the message is about. The other is deciding what should happen to it. A keyword or pattern might identify a billing issue, but routing an injury report or legal notice to a human requires a policy about consequences, not just a topic label.
I'd probably start with explicit rules for mandatory escalation, then use narrower patterns for the remaining queues, with ambiguous cases falling back to human review. That would be a deterministic baseline worth testing against the same 50 messages.
I didn't build that baseline for this experiment, so I can't claim it would perform better. But the 0/12 escalation result gives me a strong reason to investigate. Your cached-filings approach is exactly how I'd want to test the rules rather than trusting the first version.