"How much does it save?" is an easy question to answer badly. The same plugin, on the same 400-case workload, produced 86.1% fewer billed tokens at one threshold — and 8.5% at another. Both numbers are true, and neither means anything without its conditions.
This post is about the measurement behind them: why "token savings" is hard to pin down inside an agent pipeline, what we count and at which granularity, where each digit comes from, and how to rerun the whole thing on your own traffic.
The setup. dsh-phocinae is a local decision plugin for the DeepSeek Harness agent (dsh): two tools, phocinae_ask and phocinae_gate, backed by a small local model that answers typed decision questions in a single forward pass. The workload for this measurement is the published typed-decisions row set — 400 cases, 2,000 decisions — with the local model (Phocinae-Largha-150M-v1, CPU, fp32) paired against a flash-tier hosted model (deepseek-flash, temperature 0). Hosted-side token counts come from the API's own usage counters; everything else was measured with the scripts under bench/, and the raw results are committed under bench/results/. The run is a snapshot — plugin 0.2.1, dated 2026-10-08. "Savings" throughout means billed hosted tokens avoided.
Why "token savings" is hard to measure
There is no single meter. The hosted provider meters the tokens it bills you. A local decision bills nothing — but it is not free. Its tool definitions ride in every request, and each call's arguments and rendered result enter the transcript, where they get billed like any other token on the next request. "Saved" always means "not sent", and you have to be explicit about what you counted.
The escalation unit is not the decision. When escalation happens, the hosted model is called with a whole case — one state plus its five typed questions — so if any single decision in a case is unsure, the case is re-answered wholesale. Counting "decisions kept local" and counting "cases that never hit the API" are two different measurements that will give you two different percentages.
The threshold is a product decision. The plugin produces an escalate signal, not a savings guarantee. τ — the confidence below which an answer is marked unsure — is chosen by the caller, and it is the difference between the 86.1% row and the 8.5% row below. A savings claim without a τ is half a sentence.
The workload decides. Ours is a machine-readable question set, not a live multi-turn agent session. That is a limitation with a number attached; more on it at the end.
The measurement design: three counters
We count tokens in three places.
1. Per request — the fixed part. While the plugin is loaded, both tool definitions ride in every request's tool list: 473 tokens for phocinae_ask plus 212 for phocinae_gate, or 685 tokens per request. These were not estimated; they were counted with the provider's own tokenizer. And note the scope: that number is the plugin's addition to a request, not the request itself.
The approval gate is the exception. It runs as a pre-execute hook, not as a message, so it adds zero conversation tokens — whatever it costs in wall time, it never enters the transcript.
2. Per decision — the round trip. One local call costs 99 tokens of arguments out and 135 tokens of rendered result back: 234 tokens per decision in steady state. A decision that is the first in its session also carries the fixed 685, for 919 total. The fixed counter amortizes from there; how fast is the next section.
3. Per case — the unit the hosted model is actually called with. Every hosted response reports usage, and across the 400-case run those counters summed to 1,222,211 tokens (290,606 in, 931,605 out — about 76% of the bill is the model writing its answers). That is 3,055.5 tokens per case. The local side of the same run: no billed tokens, by construction. A decision is a forward pass, not a generation — the server's usage field reports zero output tokens.
The results, with the conditions attached
Read every number below as one row of a table whose columns are: workload, model pair, τ, and whether the plugin's own footprint is included. All savings here are counted in billed hosted tokens, at the case level. Accuracy is reported next to savings, because a cheap wrong answer is not a saving.
The pure local path. Every decision answered locally, nothing escalated. Against sending everything to the hosted model, the local path saves 69.9% of tokens at one decision, 90.1% at ten, and 92.3% at a thousand. It saturates near 92%, not 100% — the missing part is exactly the fixed counter: tool definitions and round trips stay in the caller's context no matter how large the session grows.
Hybrid at τ=0.4 — the row we would point you to on this workload. 27 of 400 cases were escalated (6.8%). Billed tokens: 82,499 instead of 1,222,211. Count the plugin's own footprint — 685 per request plus 234 per kept decision — and the saving is 86.1%. Accuracy: 0.7305, against 0.7085 hosted-only and 0.7320 local-only. On this workload, escalation was close to a wash. (Both accuracies come from this protocol's rendering: paired and directly comparable to each other, not the model card's standalone figures.)
Hybrid at τ=0.6 — the honest row, and the plugin's shipped default. 363 of 400 cases were escalated (90.8%). Model-only saving: 9.2%. With the plugin's footprint included: 8.5%. Accuracy also slipped to 0.7115 — below the pure-local 0.7320 on this workload, because escalated cases are re-answered wholesale and the hosted model is weaker on this workload's choice-type questions (0.6793 vs the local model's 0.7529, on 1,400 of them).
Why does 0.6 escalate so much? Because the confidence distribution is concentrated near the middle — the median decision sits at 0.6294 — and one unsure decision is enough to flag its whole case. 851 of the 2,000 decisions (42.6%) sit below 0.6; counted per case, that escalates 90.8% of cases. A threshold that sounds conservative is, on this distribution, nearly an opt-out.
Cost, at the paired model's published unit prices ($0.28 per million input tokens, $1.10 per million output): the 400-case run comes to $1.1061 hosted-only and $0.0747 at τ=0.4. A local-only run has no billed tokens.
One discipline note before the digits drift: two savings figures from the same run can still be incomparable if one counts decisions routed and the other counts tokens billed. Every number here carries its counter, its unit and its threshold. When you see a savings headline without those three labels, you don't yet know what question it answered.
What we did not measure
-
Live agent sessions. 400 machine-readable cases is a bench, not a session trace. Real sessions have long histories, more tools, more turns. The next experiment we owe ourselves: the same task list (30+ tasks) with the plugin on and off, aggregating the API's
usageand counting escalations, false blocks and task success alongside tokens. Until that runs, treat the numbers above as a calibrated bench, not a promise. - GPU latency. Every latency figure in this post is CPU, fp32, one machine: 42.6 ms for a local decision's HTTP round trip, ~21 ms mean to screen a command through the gate, 0.02 ms for an allow-listed command. A GPU figure also circulates — fp16, p50 21.0 ms on an RTX 5090. That one is the model release's published number, not a measurement of ours; we have not reproduced it on this bench — that measurement is still pending.
- Portability of τ. 0.4 is not a universal constant; it is what this workload's confidence distribution produced. Re-scan on your traffic.
Reproduce it on your own traffic
Everything above is re-derivable from bench/:
# 1. serve the local decision model
python -m phocinae.main
# 2. the token-savings run — point it at your rows and your model pair
node bench/token-savings.mjs --rows 400 --dataset ./your_rows.jsonl --model <hosted-model>
# 3. your plugin's own footprint: tool-definition tokens, round trip, gate wall time
node bench/context-cost.mjs
# 4. amortization and hybrid rows, plugin overhead included
node bench/economics.mjs
# 5. the gate alone, against the annotated command set
node bench/gate-by-mode.mjs
Sweep τ across {0.3, 0.4, 0.5, 0.6} and read accuracy next to savings: on this workload, pushing τ up bought less saving and worse accuracy. bench/cases.json holds 53 annotated commands — replace it with your own traffic to measure your gate instead of ours.
The measure of a savings claim is whether you can rerun it. Ours: 86.1% at τ=0.4 on 400 cases, plugin overhead included — and the 8.5% row at τ=0.6, reported because a knob you don't report is a knob you don't understand. Point the bench at your logs, and trust your own row over anyone's — including ours.
- Plugin:
github.com/Phocinae/dsh-phocinae—bench/in the same repo holds the scripts and raw results behind every number above - Model (Apache-2.0):
huggingface.co/Phocinae/Phocinae-Largha-150M-v1
Top comments (2)
I re-derived the rows from your numbers and everything ties except one thing I'd like to check. 1,222,211 / 400 = 3,055.5, the 69.9% / 90.1% / 92.3% amortization rows and the 92.3% ceiling (1 - 234/3,055.5) all match. The hybrid rows only match if the 234 tokens of local round trip are counted for the kept cases (373 at τ=0.4, 37 at τ=0.6) and not for the escalated ones: 82,499 + 373×234 + 685 gives 86.05%, and the τ=0.6 row gives ~8.4%.
But the escalate signal comes out of the local call, so an escalated case has already paid the 234 before the hosted model is called. Counting all 400 gives 85.5% at τ=0.4 (a small change) and about 1.5% at τ=0.6, not 8.5%. Unless the gate screens without a transcript round trip in that path, the shipped-default row is probably lower than reported, and it makes your "τ without a label is half a sentence" point stronger.
Same question for the 685: tool definitions ride in every request, so in a multi-turn session the fixed part scales with request count, not once per run. Do you have a row that charges it per request?
I hit a related measuring problem with tool context: I built canli-mcp (free, MIT, no key;
npx -y canli-mcp, github.com/arhancanli/canlicapital) and compared 306 tools behind three front tools at 1,109 tokens against 34,508 for listing them all. The fixed cost only means something next to the request count it's charged over.The decision to report the 8.5 percent row alongside the 86.1 percent row is rare discipline. Most plugin benchmarks hide the distribution and quote the boundary case.
The jump from 42.6 percent unsure decisions to 90.8 percent case escalation is a clean demonstration of serial bottleneck risk. If a case bundles five decisions and each has an independent 42.6 percent chance of flagging uncertainty, the probability that all five clear the gate without a single escalation is roughly six percent. The empirical 9.2 percent non-escalated rate suggests mild positive correlation across decisions within the same case, but the power law still dominates.
The bigger economic constraint is the all-or-nothing escalation contract. When one question flags, you absorb the local transcript overhead of tool definitions and round-trip tokens, then pay the full 3,055 billed tokens on the hosted side. That turns the local gate into an unhedged option. The setup only avoids that tax if the harness can pass the four confident decisions as pre-filled transcript and bill the hosted model solely for the marginal unsure token.