Project: Baize — an AI assistant runtime for your team (Go 1.25+, MIT)
Repo:https://github.com/rebornace/baize
This post is not a tutorial for "integrating Jev". It shares one judgment:Jev is worth learning for more than its model — the engineering idea that high-frequency decisions shouldn't ask a generative model to write an essay.
And that idea lands without depending on Jev's service at all.
1. Decision layer design: three principles + one red line
Baize is a sidecar AI assistant runtime: a single process deployed next to your business systems, turning OpenAPI / MCP / HTTP plugins into callable tools, with human approval required for write operations. It calls models through OpenAI-compatible APIs, so it cannot read logits — and therefore cannot get calibrated probabilities. The decision layer was designed from the start with no probability-dependent logic.
Three hard principles:
Pluggable — one decision interface; local rules, a local small model, or a remote decision service are all implementations. Default to the cheapest. No dependency on any external service.
Degradable — every decision point fails open to the existing rules path; the decision layer is never a single point of failure for a run.
Observable — every judgment records its source and any degradation (decide.degraded, decide.tool_narrow events).
One red line: no confidence numbers of any kind in judgment results. Without calibrated logits, a number that looks like a probability but is not calibrated is more dangerous than no number at all — it invites branches like "auto-execute when p > 0.9".
Interface shape: verdicts only
The return shape is deliberately minimal: verdicts are yes / no enums; multi-select returns the chosen items; every answer carries its source (rules, remote, or fallback) and a degraded flag. There is no float confidence field in the interface — a type-level guarantee against treating uncalibrated numbers as confidence.
The Chain: errors must be swallowed by degradation
All implementations are chained and tried in order; the first success returns. If all fail, the chain returns the fallback verdict hard-coded by the call site, marked degraded. The chain itself never returns an error — errors are consumed by degradation, never propagate into the main flow. Callers always get a legal enum value; a decision-layer failure looks like "old behavior", never like a crash.
Easy to miss: the fallback verdict is hard-coded at the call site, not a config item. Failure directions differ per decision point — memory extraction fails open to Yes (spend a call rather than lose a memory), tool narrowing fails open to the full set (spend prefill rather than miss a tool). As a config knob, someone would eventually flip it under cost pressure and silently lose data. Failure direction is a safety property, not an ops parameter.
Implementations today
Two tiers on the chain, remote first: when a dedicated decision-model profile is configured, a small model answers under a prompt that forces enum-only output, validated by regex / JSON parsing; unparseable replies abstain and let the chain degrade. Multi-select answers accept only names from the offered set; out-of-set values are dropped. Because structured-output support varies across OpenAI-compatible endpoints, the contract is enforced by prompt + parsing, not by endpoint features.
With no profile configured (or remote unavailable), the zero-latency rules backstop handles memory extraction only — it abstains everywhere else: short chitchat with no fact cue returns No; explicit fact cues ("remember that", "my number", passwords, addresses, phone numbers), a run of three or more digits, or substantial length returns Yes.
Want to plug in a self-hosted inference endpoint that can read logits later? Add one more tier to the chain; callers don't change.
2. Four decision points: moving "judgment" out of "essay-writing"
Four places in Baize were classic waste: paying a generative model to write a paragraph just to get a "yes / no / which one" answer. The decision layer moves each one down.
2.1 DP-1: memory-extraction pre-check (the best cost/benefit)
Baize used to fire an LLM call after every successful conversation to extract memory. Most conversations contain nothing worth remembering and return an empty array — but each one paid a full call plus a JSON-parse risk.
Now, before paying for that call, the engine asks the layer "is this turn worth extracting?": a No skips extraction and records a skip event (marked as decided by the layer) together with the estimated input tokens saved, so the win is measurable; a Yes — or an unavailable layer — extracts exactly as before.
The probe itself is capped at 1500 characters: the probe must never cost more than the call it tries to save.
2.2 DP-2: two-level tool-candidate narrowing (the structural one)
The most worthwhile change in Baize. Its positioning is "give it OpenAPI, get tools automatically"; the more connectors you attach, the more full tool schemas go to the main model every turn — prefill cost grows linearly with connectors. That is a structural cost that scales with your users.
Narrowing has two levels:
Level 1: system routing (System Targets). The decision model picks which backend systems — a handful of connector ids — this turn needs; it may pick several. Choosing a tool among hundreds is hard; choosing among three to five systems is easy. System descriptions are not hand-written: they are auto-derived from each system's tool names and descriptions via within-system term frequency times cross-system IDF, so adding a connector needs no routing-corpus maintenance.
Level 2: in-system keyword prefilter. Within the chosen systems, deterministic IDF keyword scoring (tool names weighted 2x, descriptions 1x, stopwords removed, CJK tokenized as bigrams), merged round-robin across systems, narrowed to 16 candidates by default.
Narrowing is not one-size-fits-all; three floors prevent over-pruning:
Category-tool floor: for "list the users", a generic pagination tool ranks poorly under IDF — but a list request needs exactly that tool. List / detail intent forces the matching admin read tool into the set.
Auth floor: once a system is chosen, its login / current-session primitives are always kept — on a 401 the model must recover the session on the spot, and the user will never say "login".
Hybrid routing: discriminative domain words in the query (e.g. pets, orders) force their systems and can never be removed by the model — the small model can only add on top, so it cannot vote away the only correct system.
Also, main-model calls support a choice constraint (tool_choice=required) via optional interfaces, without breaking existing provider signatures; providers that don't support it fail open to an ordinary call. A constraint is an optimization, not a hard gate.
2.3 DP-3: tool-result pruning
In long conversations, a big JSON returned by an earlier tool call keeps bloating the context. When compaction triggers, the layer judges each of the bulkiest tool results "worth keeping verbatim?"; unworthy ones become a short placeholder that preserves the message / tool-call pairing — the model can re-invoke the tool, and the original result stays in the run log.
Three bounds keep the judgment cheap: only tool results are judged (not ordinary turns), only those over roughly 500 estimated tokens, at most 8 per turn. All failures mean keep everything.
2.4 DP-4: tier arbitration
Baize used a pure heuristic (image present? code fences?) to route each turn to light / standard / power tiers. The problem: "help me refactor this function" contains no code fences, gets classified as light, and the strong model never runs.
DP-4 does not replace the heuristic. It only consults the layer when the heuristic lands on the ambiguous standard tier, the turn is long enough (at least 400 characters), and routing is Auto — asking for a light / power choice. Short turns are not worth asking: "hello" barely differs across tiers, and every consult costs latency.
DP-4's judgment goes through the same chain: a dedicated decision tier makes the two-way pick when available; on degradation, the heuristic's standard tier stands — the layer advises, never rewrites the routing outcome.
3. Results and validation
Reproducible benchmark: 37 real read-only business requests across 3 connected backends (390 tools total), model DeepSeek-Flash, stability run of 5 rounds (185 requests). With the default top-16: run success rate 99.5% (184/185), average turn-0 prompt ~3,090 tokens, about 34% lower than top-32; sending all 390 tools directly measured ~85k turn-0 prompt tokens. The dataset and scripts live in scripts/tool-routing-eval — reproducible.
Default narrowing: convergence kicks in above the threshold (default 12 tools); 16 candidates is the default cap — the measured cost / success sweet spot.
Regression harness: sweep (width), stability, and analyze (offline) scripts that restore your previous configuration when they finish. An optimization that changes behavior should only go live with shadow data first, then flip shadow off.
Everything ships with the decision layer off by default and enabled per point: the master switch (decide_enabled) defaults to off, each decision point has its own knob, all controlled through the existing hot-reload settings — no wiring changes.
4. Honest boundaries
No calibrated probabilities. Baize calls OpenAI-compatible APIs and cannot read logits. The layer only ever returns enums, and there is no "auto-execute above a probability threshold" feature. Write approval stays deterministic rules + human approval. That red line does not bend for Jev.
This is not "integrating Jev". The decision layer depends on no external decision service. If you later point your inference at a local vLLM running a decision plugin, the remote implementation works as-is — a free win, not an architectural prerequisite.
5. Closing
Jev made one thing click for me: models matter, of course — but plenty of high-frequency little decisions shouldn't ask a generative model to write an essay every time. Pulling "judgment" out of "generation" into a pluggable, degradable, observable layer is engineering any agent project can do — it does not depend on any company's window.
Baize's decision layer is live in the main branch (internal/decide + four decision points). Come take a look, run it, or open an issue:
More: the README's decision-layer section, and the bilingual developer docs under docs/developers
If you have hit the same wall — tool schemas going to the model every turn while prefill gets more expensive — I'd love to hear your solution. I'll keep following up on feedback and issues.
Top comments (0)