I Moved Decision Logic Out of an LLM and Measured What Broke

Two earlier posts in this series compared two "decision models" — text models that return a probability instead of prose — and found they fail in opposite ways (part 1, part 2).

But that was still a model-vs-model question. The operational question I actually cared about:

if your pipeline asks a general-purpose LLM to make a judgment call,
what actually changes when you carve that decision out
into a decision model?

pitch:      "faster, cheaper, safer"
my policy:  believe it exactly as far as I can measure it

The measurement harness and all raw data are public:

sunnydachs / decision-flow

Measure the trade-offs of moving decision logic from general LLMs to Decision Models (rule / LLM / decision model / hybrid) — same data, same metrics

decision-flow

An evaluation harness that measures the trade-offs of extracting decision handling in enterprise systems from a text-generation LLM to a Decision Model (a model that returns structured decisions), under identical data, task definition, execution conditions, and metrics.

Methods compared:

Method	Implementation	Probability handling
Rule-based	src/adapters/rule.py (keyword rules)	none (0/1 risk signal only)
General LLM	src/adapters/llm.py (prompt mode / json_schema-enforced mode)	self-reported values (not calibrated)
Decision Model	src/adapters/mercury.py, pplx_decider.py, jev.py	officially defined probabilities (calibration target)
Hybrid	src/hybrid/pipeline.py (rule → decision model → threshold → auto/review/block)	the above probabilities

The harness does not assume Decision Models are "fast, cheap, and safe". It only reports what the measured numbers show.

English | 日本語

Setup

Python 3.11+. Dependencies are split into extras.

uv venv --python 3.12 .venv
source .venv/bin/activate
uv pip install -e ".[dev]"          # pytest / numpy / scikit-learn
uv pip install -e 

…

View on GitHub

What I measured

Task: classify customer-inquiry emails into 5 categories, one of which is an urgent claim with a high cost of being missed.

The three decision models are Jev, Mercury, and pplx-decider — text models that return a probability instead of prose.

Seven approaches, all given the same data, the same category definitions,
and the same concurrency/timeout budget:

 1. keyword rules
 2. embeddings + logistic regression (local)
 3. general LLM (ask for JSON in the prompt)
 4. general LLM (JSON-schema-enforced)
 5. mercury-decide (decision model)
 6. pplx-decider (decision model)
 7. Jev (decision model)

Thresholds were tuned on a separate 300-item set only, then applied unchanged to the 500-item evaluation set.

The evaluation data itself is LLM-generated synthetic data, so treat absolute numbers as indicative and the differences between methods as the signal.

Result 1: accuracy is a tie

 keyword rules        0.732
 embeddings + LR      0.878
 mercury-decide       0.886
 general LLM (prompt) 0.908
 general LLM (schema) 0.904
 pplx-decider         0.902
 Jev                  0.900

The top five sit between 0.886 and 0.908 — within noise of each other. Neither "decision models are smarter" nor "the general LLM is smarter" survived measurement. Accuracy does not decide this comparison, so the rest of this post is about everything else.

Result 2: how much you can automate splits from 0% to 47%

This is the operational question that matters: holding recall of urgent claims at ≥95%, what share of the inbox can be auto-processed?

                      recall (95% CI low)   automation rate
 embeddings + LR       0.922 (0.885)       47.2%
 mercury-decide         0.969 (0.943)       37.8%
 pplx-decider           0.979 (0.958)       28.6%
 general LLM (schema)   0.984 (0.963)       20.2%
 general LLM (prompt)   0.990 (0.974)       14.2%
 Jev                    1.000 (1.000)        0.0%
 keyword rules          1.000                0.0%

Same safety floor, and the automatable share ranges from nothing to nearly half.

Look at Jev: zero missed urgent claims — the most careful judge in the set — yet 0% automation. The reason is the shape of its probabilities.

Result 3: "probability" means different things

Decision models return probabilities, but the distributions are nothing alike:

                    P == 0.0   P == 1.0
 Jev                  53%        24%      <- nearly binary
 pplx-decider          0%         0%      <- continuous
 embeddings + LR       0%         0%      <- continuous
 mercury-decide        0%         0%      <- extreme-leaning but never exactly 0/1

Jev returned a probability of exactly 0.0 for over half the items. Its judgments are right, but with probabilities pinned to the extremes there is no "route the uncertain ones to a human" band to operate in. Raise the threshold at all and misses appear — so the only safe operating point routes everything to review. That is what 0% automation is made of.

mercury-decide leaned extreme too (90% of its mass below 0.05 or above 0.95) — enough to cap automation at 37.8% — but it never returns exactly 0 or 1, which is why it still leaves a thin band to threshold on. This extends part 2's observation (one model binary, one spread) to 500 items — and shows it landing directly on an operational metric.

Result 4: latency and output size are a different league

                      p50 latency   output tokens / item
 keyword rules             0 ms      -
 embeddings + LR           0 ms      -
 Jev                     201 ms      53
 mercury-decide          308 ms      3
 pplx-decider            325 ms      1
 general LLM (prompt)   4133 ms      234
 general LLM (schema)   3372 ms      154

The decision models answered in 0.2–0.3 s with 1–53 output tokens per item; the general LLM took 3–4 s and wrote 154–234. "Faster" held up this time — a generation model writes prose, a decision model returns a value. One nuance: enforcing the JSON schema cut the general LLM's mean latency by 1.5 s — output-format enforcement helps speed too.

Result 5: stability differs in kind

Same input, 20 runs:

                     20 identical runs   paraphrased instructions
 general LLM (prompt)        8%                    8%
 general LLM (schema)        4%                   10%
 pplx-decider                0%                    2%

The decision model was perfectly deterministic on identical input — but paraphrasing the instructions still flipped 2% of its labels (the LLM flipped 8–10%). Decision models look less prompt-sensitive, not insensitive.

What I take from this

Stated only as far as the numbers go:

• Accuracy didn't separate the methods. 0.886–0.908 for the top five.
• Decision models were genuinely fast. 0.2–0.3 s vs 3–4 s.
• "Safe" was not automatic. A model whose probabilities collapse to 0 and 1 achieves zero misses and zero automation. A continuous-probability model automated up to 47%, but in one case let the recall floor's lower CI slip to 0.885.

So carving decisions out of the LLM is worth it on speed alone in this task — but which model's probabilities you operate on determines the shape of the whole pipeline. Before migrating, plot that model's probability distribution on your own data. That is the one check that would have predicted every operational number in this post.

The decision models also expose other question types (yes/no, score) — combining those is the next measurement.

Honest limitations

• The evaluation data is LLM-generated synthetic data with self-labels; no human double-annotation was performed. This is a relative comparison between methods, not a quality guarantee. For reference, trained human annotators agree with each other around 79% on hard classification tasks (Gilardi et al. 2023, PNAS) — all methods landing at 0.88–0.91 is at or above that band.
• One general-purpose LLM was tested. Conclusions are about that model, not about larger ones.
• Jev's stability (20 identical runs) was not measured for budget reasons.
• Costs were measured as output-token counts per item, not dollar amounts; see the repo for the per-1k estimates.
• No real customer data was used; validate with human review before production.

This is a personal OSS project — no warranty. Issues and improvement ideas are welcome.

Repo: sunnydachs/decision-flow

Comments

[ai_adam]

Thực tế mình cũng gặp case tương tự khi tách routing logic ra khỏi LLM cho một hệ thống multi-agent. Vấn đề lớn nhất không phải accuracy mà là observability — khi logic nằm trong prompt, bạn không thể trace được tại sao agent chọn path A thay vì B, cũng không thể viết unit test cho decision boundary.

Một pattern hoạt động tốt: dùng LLM chỉ để extract structured intent (JSON schema cố định), sau đó truyền vào rule engine deterministic. Cách này cho phép:

• Version control logic quyết định riêng biệt với prompt
• A/B test rule set mà không cần re-deploy model
• Debug production incident bằng replay log intent → rule evaluation

Trade-off là phải đầu tư schema design lên front — nếu schema quá rigid thì LLM hallucinate field, quá loose thì rule engine không ăn khớp. Mình hay dùng approach "progressive tightening": cho LLM tự do ở staging, collect failure cases, dần chặt schema dựa trên real data.

Curious bài viết có measure latency overhead của architecture này không? Vì thêm hop serialization/deserialization giữa LLM và rule engine đôi khi ăn mất gain từ cache hit (site: labagent .tech)

[sunnydachs]

Thank you for the observability framing - a decision boundary living in the prompt has no address, so there is nothing to unit-test. On latency, the article measures it end-to-end per item, p50: keyword rules and embeddings+LR 0 ms, decision models 201-325 ms with 1-53 output tokens, the general LLM 4133 ms prompt / 3372 ms schema-enforced with 154-234 tokens. The serialization hop is real but small: the decision path's round trip stays under a third of a second. Your progressive-tightening pattern is the right collector for what this grid leaves open - how often a too-rigid schema forces a refusal instead of a catchable field error.