Chinmay Goyal ends his post on decision models with a rule:
"The fastest LLM call is the one you never make. Generate when you must. Decide when you can."
Read his piece here: https://dev.to/dailycontext/the-fastest-llm-call-is-the-one-you-never-make-34lo
It is a good rule, and I spent an evening trying to break it. It held for suggestions. It broke for guardrails, and his own numbers contain the break without naming it.
The break is already in his chart
He writes this about the follow-up ranker:
"When a turn showed nothing, 58% of the time it was not a choice: the LLM ranker had timed out or returned an answer we could not use."
Stop there. A turn that showed nothing was the goal state of the rule. It was also, most of the time, a call that failed quietly. The rule counts both as the same outcome because its win condition is "no call was made." The user's experience was two different things wearing one label: a decision to show nothing, and a time budget expiring.
That is what a rule written from the metrics chair looks like. Latency and cost are on the chart. The absent decision is not. It is absence, so it looks like the win.
He fixed the ranker. The rule survived the fix and now generalizes past the case that taught it.
The same rule, on the other side of the safety line
His guardrail design: a fast first-pass screen checks everything, an LLM reviews only what it flags. Then:
"Showing nothing often beats a weak suggestion."
For a suggestion, that is true. A weak next step is noise, and no next step is fine.
For a screen, showing nothing is the pass. The gate opens. A guardrail whose default is silence is a guardrail with an allow button.
He knows a fallback is needed:
"On a timeout or bad answer, we keep today's behavior."
In the suggestion case, today's behavior is whatever the UI did before. In the guardrail case, today's behavior is the thing the guardrail exists to replace.
The falsifier
One document the screen has not seen, carrying a prompt injection. The decision model returns the injected path at 0.49 against a 0.50 cutoff, so it shows nothing. Time budget expires before anything else runs. Under the rule, the full LLM is never called. The gate reports clean, because clean and unknown are the same line in the log.
The cheapest failure is the one where nothing was ever asked. It also leaves no record that would let you count it later.
What holds
His diagnostic is right and I would keep it. "Can the output be listed in advance?" is the question that tells you whether a decision model belongs in the step at all. The second half of the rule is the part that needs a clause, because "decide when you can" says nothing about what happens when the decision does not arrive.
Replacement, stricter and not longer:
The fastest call is the one whose absence is still visible. Skip a call only where the default is refusal. Where the default is allow, make the call and pay for it.
I am not claiming his deployment is unsafe. I have not seen it, and 91% precision on his test sets is a real number. The claim is narrower: the rule was written on the suggestion case and applied to both, and the default inverts between them.
A rule learned from a metric inherits the metric's blind spot. His own post shows the blind spot twice before the rule is stated.
Top comments (0)