Building a local-first decision engine that only calls the big model when it's worth it

Вячеслав Барсуков — Wed, 11 Nov 2026 11:54:45 +0000

Every AI agent I've built eventually hits the same wall: calling a big model on every request is too expensive, but a small local model is too dumb on the hard cases.

Most teams pick one side and live with the consequences. I wanted a third option: run a small local model first, and escalate to a big one only when the expected benefit of escalation actually exceeds its cost.

That project is called decision-cascade. It's open source, it's on PyPI (pip install decision-cascade), and this post is about what I learned building it — including the parts that didn't work at first.

The problem with "escalate if confidence is low"

My first version did the obvious thing: if the local model's confidence < 0.6, send the request to a bigger model. Simple, works, and completely wrong in practice.

Why wrong? Because confidence and benefit are not the same thing.

You can be very unsure about a question where being wrong costs nothing.
You can be quite sure (0.8) and still be wrong on a case where being wrong costs a lot.

A confidence threshold ignores both the cost of an error and the price of the external call. So the cascade ends up escalating cheap-to-be-wrong cases and skipping expensive-to-be-wrong ones. That's backwards.

What I changed: benefit-aware escalation

The current version compares two numbers for each request:

Expected utility of the local answer — accuracy adjusted by the cost of being wrong in this particular state.
Expected utility of escalation — accuracy of the external model (estimated online via feedback) adjusted by the price of the call.

If the second is bigger than the first, escalate. Otherwise stay local.

In code, the core of it looks roughly like this (simplified):

def should_escalate(local_conf, error_cost, external_q, external_price):
    expected_local = local_conf - (1 - local_conf) * error_cost
    expected_external = external_q - external_price
    return expected_external > expected_local

The trick is that external_q (the external model's accuracy) is not a constant. It's updated online through POST /v1/feedback as users report whether the external model was right. So the cascade adapts to the actual quality of whatever external service you plug in — Clef, Laya, Perplexity, your own endpoint.

The hard gate (because small data lies)

Once I added a learned risk model P(error | state, question), things got better — until I fed it 8 examples and it overfit so badly it was worse than the original threshold.

So I added a hard gate: if the learned risk model doesn't beat a simple confidence threshold on holdout data, the cascade falls back to the threshold automatically. It's visible in the response as usage.reason = "risk_gate", so you can see exactly when it happens.

This isn't glamorous, but it's the difference between "a research prototype" and "something you can actually run in production."

Benchmarks (with a 16GB GPU, n=200)

I benchmarked against real releases of Clef-Flash (9.4B, GGUF Q4_K_M) and Laya (421M, ModernBERT-large) on two public datasets. Same hardware, same prompts, same evaluation script.

Dataset	decision-cascade 124M	Clef-Flash 9.4B	Laya 421M
AG News — accuracy	0.873	0.855	0.662
AG News — p50 latency	10 ms	130 ms	16 ms
RouterBench — accuracy	0.542	0.442	0.312
RouterBench — p50 latency	21 ms	170 ms	—

A 124M model beating a 9.4B model sounds suspicious, so let me be precise about what this does and doesn't mean:

It does mean that on decision tasks (classification, escalation decisions), a small model with task-specific heads and calibrated escalation can outperform a much larger general model.
It does not mean the small model is better at free-form reasoning or generation. Clef wins easily there.

Everything is reproducible: eval/competitors/reproduce_real.sh plus raw JSON in the releases.

Two demo games (because synthetic benchmarks aren't enough)

Synthetic datasets don't answer "does this work where mistakes have a real cost?" So I built two games — fully local, no third-party assets:

Chess vs Stockfish: 19% escalations close half the gap to "always search deep," at 38% of the full-search cost.

Pac-Man-style maze: cautious policy scores 94% of the maximum with 33% escalations — twice the greedy strategy.

The chess demo is the one I show people first. It makes escalation cost visible: you can literally watch the engine decide "this position is uncertain, but it doesn't matter — stay shallow."

Install and try it in 2 minutes

pip install decision-cascade

Fastest way to see it work, no GPU and no model downloads:

DECISION_ENCODER=hash uvicorn api.server.main:app --port 8000

This returns responses with trained: false — that's honest, the default heads are random. For real quality, either download a pre-trained pack (v0.2.0) or run SFT on your own data:

git clone https://github.com/slavabars/decision-cascade
cd decision-cascade
pip install -e ".[train]"
python -m train.sft --config configs/sft/support.yaml

The trained weights land in checkpoints/ and are loaded automatically on server start.

What's honest about the limitations

Default heads are random. No amount of clever cascade math saves you from untrained weights. You need SFT/RLCD or a weight pack.
The risk model needs examples. Out of the box it's no better than a threshold — and the hard gate admits this in the response.
Domain calibration is on you. AG News and RouterBench are public benchmarks, not your support tickets. Calibrate on your own data before trusting escalation rates.
Cold start on CPU is slow. First inference after a model load takes a few seconds. The hash encoder is the fastest way to test the pipeline without waiting for that.

Why local-first matters (for me)

Two reasons, in order:

Cost. 70–90% of decisions in a typical product are routine. Escalating all of them is burning money for no quality gain.
Privacy. Some decisions shouldn't leave the building at all. A local cascade lets you keep sensitive cases on-prem while still escalating the ambiguous ones to a stronger model — if you choose to.

The project speaks the SystemOne protocol, so it can act as a local gateway to Clef, Laya, or Perplexity Decider — or run entirely self-contained.

What I'm working on next

A transformer encoder with attention masking over multiple states (for batching many decisions at once).
Deeper integration examples with chat agents.
More demo environments where escalation has a measurable cost.

If you're building something similar, I'd love to hear how you decide when to escalate. The math I'm using is not the only reasonable one, and I'm curious what trade-offs other people made.

Repo: github.com/slavabars/decision-cascade

DEV Community: Вячеслав Барсуков