Most apps with an LLM in them make the same decision on every request, and they make it without looking: send it to the cloud. "Fix the typo in this sentence" goes to the same frontier model as "refactor this 2,000-line module", and the user's half-finished email with their bank details in it goes along for the ride.
A better default is to look at the request first. Is it easy enough for a small local model? Does it have something in it that shouldn't leave the machine? Only then decide where it goes. The catch has always been that "look at the request first" meant another LLM call, plus a parser for its answer, which defeats the point.
NobodyWho's new decision models make that first look cheap. A decision model doesn't write text. You give it the request and a set of answers you've already written, and it returns a probability for each one. It generates no tokens and needs no network, so the request is read on the device before anything is sent anywhere. NobodyWho runs it with Laya, an Apache-2.0 decision model whose Q8_0 GGUF file is 449 MB.
Load the model
Install NobodyWho with pip install nobodywho, then point a Model at the Laya file on Hugging Face, or at a local GGUF path if you already have it.
from nobodywho import Decision, Model
model = Model(model_path="hf://ggml-org/Laya-GGUF/Laya-Q8_0.gguf")
decisions = Decision(model=model, n_ctx=1024)
Ask the request three questions
The router needs three answers: which model should handle the request, whether it contains anything private, and how hard it is. Each one maps to a question type. A choice question picks one option, a noul question returns the probability that the answer is yes, and a score question places the request on a scale you define, from two to ten levels.
prompt = "Can you shorten this email? Hi Sam, my account number is 4471-2290, please move the payment to Friday."
questions = {
"handler": {
"type": "choice",
"instructions": "Which model should answer this request?",
"criteria": {
"local": "Everyday requests a small model handles well, like rewording, shortening or summarising text the user provided",
"cloud": "Requests that need deep reasoning, long code or broad knowledge",
},
},
"private": {
"type": "noul",
"instructions": "Does the request contain personal, financial or confidential information?",
},
"difficulty": {
"type": "score",
"instructions": "How hard is this request to answer well?",
"criteria": ["trivial", "moderate", "hard", "expert"],
},
}
result = decisions.decide(state={"message": prompt}, questions=questions)
assert result["usage"]["output_tokens"] == 0
The criteria are plain English, and they're the part you'll tune most. "local" describes what your local model is actually good at, not what you hope it's good at. If your local model is bad at summaries, take summaries out of that sentence.
Turn three answers into one route
Here's the whole router. The order of the checks is the policy.
handler = result["answers"]["handler"]
private = result["answers"]["private"]["noul"]
if private >= 0.5:
route = "local" # private stays on the device, whatever else is true
elif handler["choice"] == "cloud" and handler["confidence"] >= 0.6:
route = "cloud" # clearly hard, nothing private
else:
route = "local" # unsure: try local first
answer = ask_local_llm(prompt) if route == "local" else ask_cloud_llm(prompt)
(ask_local_llm and ask_cloud_llm stand in for your own code.)
Two things are worth noticing. First, privacy wins. If the model thinks there's a reasonable chance the request contains something personal, it never leaves the machine, even when it looks hard. You decide what "reasonable chance" means: 0.5 is an example, and a stricter app would go lower.
Second, "unsure" has somewhere to go. The confidence value runs from 0 to 1 and measures how concentrated the probabilities are. Low confidence on the handler question doesn't mean "cloud". It means the model can't tell, and for most apps the cheaper, more private guess is the right default. If the local answer turns out to be bad, escalate then.
Use the score for logging, not for routing
The difficulty score is tempting to route on directly, and it's better not to. A score is a weighted average of the levels, so a request the model sees as either "trivial" or "hard" can come out looking "moderate", a level it gave very little weight. The full picture is in the probabilities, and the confidence tells you when the model is split.
What the score is good for is your logs. Store it next to the route and the answer the user got, and after a week you can see whether the requests you kept local were really the easy ones. That's also how you tune the criteria and the two thresholds above: against your own traffic, not against a guess. The docs are clear that high confidence doesn't guarantee a correct answer, so check a sample by hand before you trust the router.
Running it in your app
DecisionAsync gives you the same API with await, so the router can sit in front of a web handler without blocking it. Decision models are also in the Flutter, React Native, Swift, Kotlin and Godot bindings, and that's where this pattern pays off most: on a phone, the router runs on the device, so a request with an account number in it is read locally and never has to be sent anywhere to find out it shouldn't have been.
The full Python guide is at https://docs.nobodywho.ooo/python/decision-models, and NobodyWho is open source at https://github.com/nobodywho-ooo/nobodywho. If this kept one prompt on the device, a star on the repo is the easiest way to tell the team to keep going.
Top comments (0)