Something interesting appeared in a recent LocalLLaMA project showcase. Several projects were exploring small models that do not generate prose. They take a state, a question, and a fixed set of choices, then return a probability for each choice.
That sounds less impressive than an autonomous agent narrating its plan. It may also be exactly what many agent loops need.
Consider the decisions inside a typical tool-using workflow:
- Which tool should handle this request?
- Is this command safe enough to run automatically?
- Did the previous step succeed?
- Should the workflow retry, stop, or ask a person?
- Which retrieved document should be examined next?
These are closed-set decisions. Asking a generative model to produce a paragraph, parsing that paragraph, and hoping it stayed inside the allowed choices is a surprisingly elaborate way to select one value from an enum.
My current view is simple: use generative models for open-ended reasoning and content. Use constrained decision components when the answer space is already known.
Generation and decision are different jobs
A generative model is useful when the output cannot be enumerated in advance. Writing a migration plan, diagnosing an unfamiliar failure, explaining a trade-off, or proposing a patch all need flexible output.
Routing a ticket to one of twelve queues does not.
We often use the same large model for both jobs because the API is convenient. The model receives a prompt such as:
Choose one action: retry, stop, escalate.
Return JSON only.
The application then validates the response, repairs malformed JSON, rejects invented actions, and perhaps asks the same model to try again. We added a language generator, then built a fence around its language generation.
Structured output improves the interface, but it does not change the underlying job. The system still generates tokens to choose from a fixed set.
Projects such as Contrastive Language Models approach the problem differently. CLM encodes the current state and candidate actions, scores their relationship, and returns a distribution over the supplied actions. The repository describes uses such as tool routing, trajectory ranking, and retrieval shortlisting.
Decider takes another route. It reads a state and typed questions, then produces probabilities for fixed choices, yes-or-no decisions, or described score levels in one pass. Its core promise is deliberately narrow: no free-form decoding and no output outside the choices supplied by the application.
These projects are early and their benchmark claims need independent validation. The architecture is still worth paying attention to.
The application should own the choice set
In a production agent, the model should not decide which actions exist. The application should.
application state
+
allowed actions
↓
decision component
↓
probabilities
↓
policy and threshold
↓
execute, ask, or stop
That boundary gives us several useful properties.
First, impossible actions stay impossible. If delete_customer is not in the candidate set, the decision component cannot select it. This is stronger than telling a model in a prompt not to mention deletion.
Second, the result is inspectable. A distribution such as this is easier to evaluate than a persuasive paragraph:
{
"retry": 0.18,
"stop": 0.07,
"escalate": 0.75
}
Third, the application can apply policy after inference. A high score is evidence, not authority.
if decision == "retry" and probability >= 0.90 and retry_count < 2:
retry()
else:
request_review()
The model proposes. Code decides what that proposal is allowed to do.
Probability is useful only when it is calibrated
A number that looks like confidence is not automatically a trustworthy probability.
If a model assigns roughly 0.8 to one hundred decisions, we would like about eighty of them to be correct. That is calibration. A model can have good top-choice accuracy and still be overconfident, which makes threshold-based automation dangerous.
This is where a typed decision interface improves the engineering conversation. We can measure:
- accuracy by decision type
- false-allow and false-deny rates
- calibration error
- coverage at a chosen threshold
- latency and cost per decision
- performance when none of the supplied choices is correct
The last case is easy to miss. A closed set can be wrong. If the workflow offers retry, stop, and escalate, but the correct action is refresh_credentials, the model cannot repair the application's incomplete choice set.
Include an unknown or none_of_the_above option where appropriate. More importantly, evaluate whether it is selected when the input falls outside the known cases.
The threshold also depends on consequence. A 0.75 routing decision may be acceptable for choosing a knowledge-base category. It is not enough evidence to run a destructive infrastructure command.
Guardrails belong around the tool, not inside the prompt
The decision model is not a security boundary.
The OpenAI Agents SDK guardrail documentation makes a useful separation: tool input guardrails run before execution, tool output guardrails run after execution, and either can reject content or halt the run. The SDK also documents important coverage limits for different tool types.
The exact framework is less important than the placement. Authorization and safety checks must sit on the execution path.
model chooses candidate action
↓
schema validation
↓
authorization check
↓
risk policy
↓
human approval if required
↓
tool execution
↓
result validation
Do not ask the model, "Is the user allowed to delete this resource?" when the target system can answer that question from its own permission model. Do not treat a 0.99 probability as permission. Use identity, authorization, resource state, and explicit policy.
A decision component can help classify risk or select a route. It cannot grant authority the caller does not have.
Small decisions can make large agents easier to operate
Agent diagrams tend to focus on the large reasoning step. Production failures often happen in the small transitions around it.
Should the agent retry after a timeout? Was the tool response complete? Does the evidence support the conclusion? Is the next operation reversible? Those decisions determine whether an isolated model error becomes a repeated side effect.
Breaking them out creates observable control points:
request
↓
route decision
↓
reasoning or retrieval
↓
action decision
↓
policy check
↓
tool
↓
success decision
↓
continue or stop
Each decision can have its own dataset, threshold, fallback, and owner. That is less magical than one agent prompt. It is also much easier to test.
This direction fits the broader advice in Anthropic's Building effective agents: start with the simplest workable pattern, prefer predefined workflows when the path is known, and add agent autonomy when flexibility justifies the added cost and risk.
The interesting part is not replacing every generative call with a classifier. It is noticing which calls were never generative problems in the first place.
Evaluation needs real workflow data
A benchmark can tell us whether a model distinguishes choices in a prepared dataset. It cannot tell us whether those choices represent the messy states in our system.
I would build the evaluation set from production-shaped examples:
- normal cases that should pass automatically
- ambiguous cases that should request review
- rare failures that previously caused incidents
- adversarial input that tries to steer the decision
- stale or incomplete context
- cases where the correct choice is absent
- repeated attempts after a tool failure
Label them with the decision that the workflow should take, not the explanation we hope the model writes.
Then compare at least three baselines:
- Deterministic rules.
- A general generative model with structured output.
- A specialized or constrained decision model.
Rules may win. For example, retry_count >= 2 does not need inference. A model becomes useful when the state contains language or evidence that code cannot classify reliably with a small, stable rule set.
Run shadow evaluations before granting automation. Record the model's decision and probability without executing it. Compare that output with the actual operator or workflow decision. This shows where a threshold would automate safely and where it would merely automate confidence.
Do not create a second platform by accident
There is a predictable failure mode here. A team sees specialized decision models, creates a separate serving stack, a feature store, a training pipeline, an evaluation service, and a new control plane before proving that one decision benefits from any of it.
Start with one bounded choice that occurs frequently enough to matter.
Tool routing is a reasonable candidate. Define the allowed tools, create a representative test set, compare rules and models, and measure error cost. Serve a smaller model only if it produces a material latency, cost, privacy, or accuracy benefit.
For low-volume workflows, a general model with strict structured output may remain the simplest operational choice. Fewer services can be worth a little extra inference cost.
The recommendation changes when decision calls dominate the loop. If an agent makes dozens of small choices per task, generation latency and token cost accumulate. A fast decision component can then reduce both, while giving the application a consistent probability interface.
Keep the agent for the parts that need an agent
Open-ended work still benefits from capable generative models. A decision model will not investigate an unfamiliar production failure, compare architecture options, or write a useful migration plan.
It can decide which diagnostic to run next from an approved list. It can estimate whether the evidence supports another step. It can route uncertainty to a person before the workflow turns a weak guess into an action.
That is enough.
An agent does not become less intelligent because some decisions move into smaller, constrained components. The system becomes clearer about where intelligence is needed, where policy belongs, and where ordinary code is still the better tool.
If the answer must be one of five choices, make the system choose one of five choices. Save the essay for the part that needs an explanation.
Top comments (1)
Our ticket router had this problem last year where overall 92% accuracy hid that the most confident refund calls were wrong more often than the mid-confidence ones. Saw it in a RAG pipeline I was debugging too, where the model was certain on bad retrievals and hesitant on the correct ones. We fixed it by logging everything above 0.85 before it could execute, which took embarrassingly long to decide we needed.