Ollaya Ollama Decision Model Setup [2026]: Routing + Benchmarks

Originally published at kunalganglani.com — read it there for inline code, hero image, and live links.

You’re going to finish this guide with a local Ollaya server running on your laptop, a real questions.json router you can drop into an agent, and a benchmark harness that reports p50/p95 latency (warm vs cold) for decision-model routing vs “LLM-only JSON routing”.

If you already have an agent stack that does tool selection with a full LLM call, this is the fastest way I know to shave hundreds of milliseconds and a chunk of cost off every request.

The target keyword here is ollaya ollama decision model setup. I’m going to give you the minimal setup first. Then we’ll get into the stuff that actually matters in production: thresholds, abstain/escalate policy, and what you should log so you can debug the inevitable “why did it pick that tool?” incident.

What is Ollaya

Ollaya is a local server for running Jev-style decision models that answer typed questions with calibrated probabilities in a single forward pass and never generate text. You send it a “state” (text or JSON) plus a question schema, and it returns typed answers, confidences, and per-option probabilities.

Routing and tool selection are basically classification problems. Using a full LLM to emit JSON for “which tool should I call?” works, but it’s usually the wrong tool for the job. You’re paying for token-by-token generation when what you want is: pick an option and tell me how sure you are.

On Ollaya’s homepage, a five-question request to the laya router model is reported at ~8–10 ms end-to-end via HTTP on an RTX 4090, versus 236–276 ms median for the hosted TypeSafe Jev API in third-party benchmarks. Even if your hardware is slower, you’re still often looking at an order-of-magnitude win.

One important 2026 reality check: Ollaya does not replace Ollama (or any text model runtime). It’s not trying to. Per the API docs, Ollama-style text endpoints like /api/generate, /api/chat, and /api/embed return 404 because decision models don’t generate text. You pair Ollaya with whatever you use for the “real” model call.

Install Ollaya / Quickstart

Ollaya’s quickstart is intentionally boring. That’s a compliment. Boring installs ship.

• Linux/macOS:
  • curl -fsSL https://ollaya.dev/install.sh | sh
• Windows (PowerShell):
  • irm https://ollaya.dev/install.ps1 | iex

The quickstart notes that on Linux and Windows it will also fetch CUDA libraries if it detects an NVIDIA GPU. On systemd-capable Linux it can set up a service bound to 127.0.0.1:11435.

Once installed, the server listens on:

• http://localhost:11435

And it exposes two API surfaces:

• Native /api/* endpoints for decision + model management
• TypeSafe-compatible /v1/* endpoints (/v1/systemone, /v1/models)

Authoritative reference: Ollaya API reference and Ollaya Quickstart.

Quickstart checklist (do this first)

1. Install Ollaya using the script for your OS
2. Run a model once (this will start the server and pull weights)
3. Hit / or /api/version to confirm liveness
4. Send a tiny /api/decide request
5. Turn on verbose timings with --verbose

Run a model (ollaya run)

ollaya run is the “stop reading docs, show me the thing working” command. It starts the server if it isn’t running, pulls the model on first use, and loads it.

Try this:

• ollaya run laya --preset triage "I was charged twice for my subscription this month and want a refund."

From the quickstart:

• laya is a router model. It routes English to laya:en and other languages to laya:multilingual.
• --preset NAME chooses built-in question sets: triage, email, guard, moderation, router, agent.
• --verbose prints per-option probabilities plus timings.
• --format json prints the full API response.

Numbers that matter operationally (because you’ll end up budgeting latency, not vibes):

• The homepage shows decider:2b answering in 178 ms on an RTX 4090 for a real example request.
• It also shows a median latency chart for multiple models, e.g. laya:multilingual 8.1 ms, laya:en 9.6 ms, decider:0.8b 155 ms, decider:2b 190 ms (hardware and precision details are on the homepage).

API basics & conventions (JSON, snake_case, limits)

If you’ve used Ollama’s API, Ollaya will feel familiar on purpose.

Key conventions from the API reference:

• Request/response bodies are JSON objects.
• Field names are snake_case.
• Requests are limited to 8 MiB.
• Unknown request fields are ignored; null means absent.
• Model names are case-insensitive and canonicalized in responses (e.g. laya:latest).
• Probabilities/confidences are rounded to 4 decimal places.
• Durations are nanoseconds; timestamps are RFC 3339 UTC.
• /api/pull and /api/create stream newline-delimited JSON unless you set "stream": false.
• Every response includes X-Request-Id. /v1/* also includes x-typesafe-request-id.

This sounds pedantic until you’re staring at logs at 2 a.m. and realize you can’t correlate anything because request IDs weren’t captured.

Ask your own questions (questions JSON)

This is where Ollaya stops being a cool demo and starts being useful.

You define a question schema as JSON. From the quickstart, question types include:

• choice (pick one of N criteria)
• score (pick a score along a criteria list)
• noul (binary/yes-no style)

A practical routing schema for agents is usually 5–8 questions. Past that, you’re doing “analysis”, not routing. And analysis belongs in the big model call, not your fast-path router.

Here’s a routing-oriented question set I like because it builds in abstain/escalation instead of pretending the model is always confident:

• tool (choice): which tool category to use
• risk (score 3): low / medium / high
• needs_privileged_action (noul): does this require credentials or stateful access
• is_user_request_clear (noul): can we act without asking a follow-up
• should_escalate_to_llm (noul): the cheap model admitting it’s not sure

Run your own questions via CLI:

• ollaya run laya --questions questions.json "..."

Or via the API:

• POST http://localhost:11435/api/decide

Per the API docs, /api/decide is also the endpoint that can load/unload a model depending on request parameters.

Safe thresholds (don’t YOLO the probabilities)

Calibrated probabilities are the point. But you still need a policy layer that turns numbers into behavior.

My default playbook for tool routing:

• If the top choice probability is >= 0.80, route directly.
• If it’s 0.60–0.80, route but add guardrails (extra validation, narrower tool args).
• If it’s < 0.60, abstain and escalate to an LLM router call.

Yes, those thresholds are arbitrary. That’s fine. The mistake isn’t picking numbers. The mistake is picking numbers and never checking if they match reality.

When I built the Walmart conversational commerce chatbot at Firework (Zealsight), handling millions of queries daily at sub-second response times, the pattern was brutally consistent: retrieval quality and routing quality dominated perceived answer quality more than swapping one model for another. The cheapest wins were almost always in the “decision layer”, not the “generation layer”.

Use an existing TypeSafe client (TypeSafe compatibility)

Ollaya’s killer feature is that it “speaks TypeSafe”. It exposes:

• POST /v1/systemone
• GET /v1/models

…with request/response shapes compatible with TypeSafe’s SDK.

From the homepage and docs: the official TypeSafe Python SDK 0.7.1 works unchanged against a local Ollaya server.

Set these env vars:

• TYPESAFE_BASE_URL=http://localhost:11435
• TYPESAFE_API_KEY=local (any non-empty value works unless you configure OLLAYA_API_KEY server-side)
• TYPESAFE_DEFAULT_MODEL=laya (otherwise the SDK uses its default)

Authoritative reference: TypeSafe compatibility · Ollaya.

Two operational gotchas called out in the docs:

• The SDK times out after 10 seconds and retries.
• The first request may include model-load time. If the load continues after the request times out, the retry might hit a warm model. That can make naive benchmarks lie.

If you’ve ever benchmarked something once, posted the chart, and then wondered why prod didn’t match. This is how that happens.

Bake your questions into a model (Modelfile + ollaya create)

If you’re going to use the same question set everywhere, don’t ship questions.json through five services and hope it stays in sync. Bake it into a derived model.

Ollaya’s Modelfile supports:

• FROM base model (can be a router like laya)
• QUESTIONS inline JSON or file path
• CALIBRATION refit temperatures (more below)
• PARAMETER precision fp16|fp32 (pin precision)
• DESCRIPTION, LICENSE

Example from the docs:

• ollaya create triage -f Modelfile
• ollaya run triage "I was charged twice for my subscription this month."

Authoritative reference: Modelfile · Ollaya.

Calibration workflow (the part people skip, then regret)

Ollaya calibrates with temperature scaling. In plain English: it rescales logits so the probability values behave more like real-world confidence.

The Modelfile CALIBRATION directive lets you replace base temperatures with refit ones based on your labeled data. This is exactly what you want if you’re going to treat thresholds like 0.80 as a contract.

My opinionated production workflow:

1. Log state + model answers + probabilities for every routing decision.
2. Sample 200–1,000 decisions per route type and label the correct route.
3. Refit calibration temperatures.
4. Validate calibration with metrics like Expected Calibration Error (ECE) or Brier score.
5. Only then lock in thresholds.

If you don’t do steps 2–4, you’re treating “0.83 confidence” like it means something universal. It doesn’t. It means “0.83 under whatever distribution you trained on, plus whatever drift you’ve already accumulated.”

CLI reference essentials (run/serve/pull/list/ps/show/stop/rm/create)

The docs have a full CLI page. In real life you mostly need these:

• ollaya serve (run the server)
• ollaya run <model> (start server if needed, pull/load, ask questions)
• ollaya pull <model> (download weights)
• ollaya ps (what’s loaded in memory)
• ollaya tags or GET /api/tags (what models exist locally)
• ollaya show <model> or POST /api/show (model details)
• ollaya stop <model> (unload)
• ollaya rm <model> or DELETE /api/delete (delete)
• ollaya create <new> -f Modelfile (derived model)

I strongly recommend scripting model pulls in CI for any environment that autos-scales. “No implicit pulls” is a great design choice, but it also means your first request in prod won’t magically fix missing weights.

Routers (laya routing behavior) and model choices

Ollaya’s homepage puts it plainly:

• laya is the fastest.
• decider is more accurate.

The router behavior you should internalize:

• laya can return model: laya:en or laya:multilingual depending on the text. Language routing becomes basically free, which is exactly how it should be.

A practical selection guide:

• Use laya for high-volume intent/tool routing where you can tolerate occasional abstain-and-escalate.
• Use decider when the decision is higher-stakes and you’d rather pay 150–200 ms locally than risk a misroute.

This is the cascade agent stacks should be doing by default. Cheap, local, deterministic-ish decision first. Expensive model call only when you have to.

If you want more background on why this routing layer matters in agent stacks, I’d read my own post on AI agents and the deeper production angle under AI in production.

Benchmark harness: p50/p95 latency + cost per 10k routings

Most docs stop at “it’s fast”. That’s not good enough. You need a harness you can run in 15 minutes that answers, “Is this actually faster on my box, for my payloads, with my cold-start behavior?”

What to benchmark

Benchmark two routers:

1. Ollaya decision-model routing via POST /api/decide or POST /v1/systemone
2. LLM-only routing where you call your normal text model and ask it for structured JSON/tool selection

Report:

• p50 and p95 latency
• cold-start vs warm
• batch size 1 (routing is almost always per-request)
• payload size (stay under the 8 MiB Ollaya limit)

Methodology (apples-to-apples)

• Warm-up with 20 requests before measuring warm p50/p95.
• For cold starts: unload the model between requests (or restart the server) and measure 10 runs.
• Measure HTTP overhead: run a “noop local endpoint” to estimate baseline latency on your machine.

I keep a local benchmark database for this site at kunalganglani.com/llm-benchmarks. The pattern that keeps showing up is that local inference bottlenecks shift from “can it load?” to “what’s the steady-state throughput and tail latency?” Ollaya routing sits in a sweet spot because it’s small enough that even tail latency can be excellent.

A compact benchmark table you can fill in

Router approach	Where it runs	Typical p50 you should expect	Typical p95 risk	Cost model
Ollaya laya decision model	local (localhost:11435)	10–50 ms (GPU), 50–250 ms (CPU, ballpark)	cold start + load	$0/token, just compute
Ollaya decider decision model	local	150–250 ms on fast GPUs (per homepage: 155–190 ms on RTX 4090)	higher variance if fp32 + load	$0/token
LLM JSON router	hosted API	200–800 ms+ depending on model/region	network + model queueing	per-token + retries

The one number we can cite precisely from Ollaya’s own published data: Laya 8–10 ms median end-to-end on RTX 4090, and TypeSafe hosted Jev 236–276 ms median in third-party benchmarks called out on the homepage.

Cost per 10k routings (rough but useful)

If your current router call is a small hosted LLM request, you often pay for:

• prompt tokens (system + tool descriptions)
• output tokens (JSON)
• retries

If you don’t already do this math, use my LLM cost approach. Or pull current token pricing from the tracker I maintain at kunalganglani.com/llm-prices to compute your own “10k routings” number.

Even if your LLM router is “only” $0.001 per decision, that’s $10 per 10k routings. At scale, routing becomes a line item. Decision models make it disappear.

Model management endpoints, errors, and deployment gotchas

Model management endpoints

From the API reference, you’ll use these a lot:

• GET /api/tags (local models)
• POST /api/show (one model details)
• GET /api/ps (loaded models)
• POST /api/pull (download)
• DELETE /api/delete (remove)
• POST /api/copy (copy)
• POST /api/create (derived model)
• GET /api/version (server version)

Errors (codes, error body)

Every error response follows:

• { "error": "...", "code": "..." }

The docs explicitly say: don’t parse the human-readable error message. Branch on code.

If you’re integrating into an agent stack, treat these codes differently:

• MODEL_NOT_FOUND: operational misconfig. Fix the deploy pipeline.
• INVALID_REQUEST: developer bug. Fix the client.
• timeouts: capacity or cold-start. Add warm pools.

Security considerations

Ollaya binds locally by default (the quickstart mentions 127.0.0.1:11435). Keep it that way unless you have a real reason not to.

If you must expose it:

• Put it behind a reverse proxy
• Require an API key (OLLAYA_API_KEY is referenced in TypeSafe compatibility docs)
• Enforce request size limits (Ollaya already caps at 8 MiB)
• Log request IDs and caller identity

If you’re using this in an agent that can be attacked, assume you’ll see weird inputs. I’d pair this with the controls in my AI security guide and, for tool-use stacks, the prompt injection threat model.

Using Ollaya with MCP/agents for tool selection

Ollaya has an “Agents (MCP)” section in its docs nav. The pattern is straightforward:

1. Run Ollaya locally for routing
2. Use the decision output to select which MCP tool (or tool group) is allowed
3. If confidence is low, ask the LLM to decide with more context, or ask the user a clarification question

This keeps your fast path fast. It also forces you to be honest about ambiguity. Some requests really do need more context than a five-question router can see.

If you’re building tool-heavy systems, read this alongside my post on agent orchestration and the protocol comparison in MCP vs OpenAI Function Calling.

Here’s the challenge I’ll leave you with: instrument your router like it’s production software, not magic. Log per-question probabilities. Track abstain rates. Track misroutes. Then tune thresholds like you’d tune a circuit breaker.

The teams that win with agents in 2026 won’t be the ones with the fanciest model. They’ll be the ones who can tell you, with a straight face and a chart, why their system picked Tool A at 11:03:12 and Tool B at 11:03:13.

Here’s a good jumping-off point video if you want more context on Jev-style models:

[YOUTUBE:mJnOdnOrh9A|Jev Is Here, But How Do You Use It?]

---

Originally published on kunalganglani.com