Can a Decision Model Replace Hand-Written Routing Rules? A 3,080-Message Experiment

Mo Al Mamun (Mamun) — Sat, 17 Oct 2026 16:45:02 +0000

A customer writes, “My card still hasn't arrived.” Somewhere in the backend, that sentence has to become a concrete choice: track an existing delivery, explain delivery times, or order another card.

I wanted to test whether a decision model could replace the hand-written logic that makes that choice.

So I built switchless: a fictional banking backend with 77 actions, four interchangeable dispatchers, and an evaluation on the 3,080-message Banking77 test set.

The result was useful, but less tidy than the original hypothesis. JEV beat our rules and answered about nine times faster than the LLM. The LLM was cheaper. At the same number of referrals, combining rules and JEV produced fewer incorrect automatic selections in an exploratory comparison, but still left roughly one wrong choice in eleven.

Disclosure: This is an independent experiment. I have no relationship with TypeSafe and received no credits, funding, or other support from them for this work.

The part of the backend I wanted to replace

This experiment is about interpreting language into an action. It isn't about replacing a gateway's endpoint table, authentication checks, permissions, or load balancing.

For an explicit request such as POST /cards/123/freeze, the destination is already known. For “I can't find my card,” the application has to interpret the request first.

The fictional product, Kestrel Bank, has 11 service modules and 77 implemented handlers. Some return information; others change an in-memory bank's state. They run inside one application, rather than as independently deployed microservices.

Customer message
      |
      v
POST /help
      |
      v
Interchangeable dispatcher
      |
      +--> Selected action --> Existing handler
      |
      +--> Abstain --> Escalation response / suggestions

Keeping the handlers and gateway shared let us change the selector without rebuilding the rest of the application.

Four ways to make the same choice

Rules: an ordered keyword-and-regex dispatcher, developed in eight stages. It abstains when no rule matches.

JEV: TypeSafe's jev-1.13.0, asked to choose among the 77 action IDs and their one-line descriptions.

LLM: OpenAI gpt-6-luna, given the same catalogue and customer message, with structured output and reasoning disabled.

Hybrid: run both rules and JEV. Accept their selection when they agree. When they disagree, accept JEV only if its confidence is at least 0.9; otherwise abstain.

The two model routers received no example messages in their requests. Here, “zero-shot” describes that setup; it doesn't establish whether either provider's training data included this public benchmark.

The rules were written by an AI coding assistant and reviewed as part of the project. That matters: this is a comparison against one disclosed implementation, not the best rule system a specialist team could build.

How the experiment was measured

Banking77 contains short banking-support messages labelled with 77 intents. Each intent maps to one action in Kestrel Bank.

We reserved 1,000 training messages as a held-out measurement sample and used the other 9,003 for rule development. A different 1,000-message sample, drawn from the rule-development pool, was used to choose the hybrid strategy and threshold. It was disjoint from the held-out measurement sample, but overlapped the data available for rule development. The final test evaluation used all 3,080 test messages. Rules, JEV, and hybrid settings were frozen before it; the LLM was added afterward in a separate run without retuning those systems.

The main metric was exact-action accuracy. Sending a message to the right service but choosing the wrong handler did not count as correct. Abstentions count as misses in overall accuracy and are reported separately.

The results

Measure	Rules	JEV	LLM	Hybrid
Exact-action accuracy	65.5%	80.9%	81.5%	74.1%
Referred / abstained	17.0%	0%	0%	18.3%
Accuracy on answered messages	78.9%	80.9%	81.5%	90.7%
Median decision time	0.29 ms	111 ms	975 ms	112 ms
API cost per 1,000	$0	$0.083	$0.028	$0.083
Router implementation lines	772	60	95	888

These are recorded-run measurements. API cost excludes infrastructure and human work. The hybrid's line count includes both underlying routers and its combining logic; the model integrations don't include provider-side model complexity.

JEV outperformed this rules implementation in each of the 11 service modules.

More rules helped, until the gains became small

On the 1,000-message held-out sample, the first 137 lines reached 39.4% accuracy. At 380 lines, accuracy was 55.2%. The final 772-line version reached 63.0%.

Later stages often repaired earlier rules. A concrete example: a pattern for cheque|check also caught the verb “check,” confusing a request about checking card status with a cheque-related intent.

The append-only development policy made the history inspectable, but it also inflated the accumulated code compared with normal maintenance, where an engineer could edit or remove an old rule. The curve describes this development process, not a universal ceiling for rules.

The effort curve uses the separate 1,000-message held-out sample. Lines measure local implementation size, not engineering hours or provider-side model complexity.

JEV was faster, while the LLM was cheaper

JEV's median decision took 111 ms, compared with 975 ms for Luna: about an 8.8-fold difference in these runs.

Accuracy was close. The LLM's observed advantage was 0.65 percentage points, with a paired 95% confidence interval from −0.38 to +1.68 points. No difference was detected on this test split; that does not prove equivalence. On the separate 1,000-message held-out sample, the LLM led by 2.20 points, with an interval from +0.19 to +4.21.

The price result contradicted the original expectation. The LLM cost about a third as much. Its repeated action catalogue benefited from cached-input pricing. Cheap classification isn't automatically cheaper than a low-cost generative model under a particular workload.

JEV was pinned to jev-1.13.0; the LLM had no pinned snapshot and was run once per evaluation split. On a fixed 200-message test subset, two repeat JEV runs changed 2.0% and 0.5% of selections relative to the original run; reversing the option order changed 5.0%.

The intervals describe uncertainty over messages. They don't cover model drift, different network conditions, or variation across repeated API runs.

Latency uses a logarithmic scale. The observed JEV–LLM accuracy difference was within the paired 95% confidence interval.

What the hybrid actually buys

The hybrid answered 2,516 messages and referred 564. Of the answered messages, 233 were incorrect: 90.7% accuracy at 81.7% coverage.

To put that beside the single-model routers, the analysis also referred each model's 564 least-confident messages. Confidence ties were broken by dataset index, with a sensitivity calculation for other tie orders.

Policy	Referrals	Accuracy on answered (95% CI)	Incorrect automatic selections
Hybrid	564	90.7% (89.5–91.8)	233
JEV, confidence-gated	564	88.8% (87.5–90.0)	282
LLM, confidence-gated	564	87.7% (86.3–88.9)	310

Intervals are 95% Wilson intervals for each policy’s answered accuracy, conditional on the selected messages; they do not account for selecting cutoffs on the test set. They are not intervals for the differences between policies.

This is an exploratory comparison at equal referral volume. The model cutoffs were selected on the test set to match the count, while the hybrid threshold was fixed beforehand. These policies refer different messages, which may require different amounts of human effort. Reordering boundary ties moved the LLM's answered accuracy between 87.3% and 88.1%; it did not reverse the observed ordering.

The hybrid made fewer incorrect automatic selections in this comparison; this observed difference does not establish superiority on future traffic. It also retained all the rules and called JEV on every message. That is a tradeoff, not a free improvement.

JEV’s threshold sweep is exploratory and uses the test split. Referrals are abstentions; human resolution and actual review effort were not measured.

Correct selection and safe execution are different questions

The gateway executes its selected handler immediately. Twenty handlers can change bank state, including blocking a card, opening a claim, or cancelling a pending transfer.

Of the hybrid's 233 incorrect selections, 32 selected a handler capable of changing state. These are not 32 observed harmful actions: whether a handler changes anything depends on the customer's account state, and the evaluation logs record predictions rather than real customer outcomes.

Likewise, an abstention only produces an escalation response or suggestions. We did not measure a human successfully resolving it.

This distinction changed how I describe the project. The results support replacing some hand-written intent-selection logic. They do not establish that the resulting actions are safe to execute without further checks or confirmation.

Higher confidence corresponded to higher accuracy, but did not guarantee correctness. Bins include at least 20 messages; calibration of intent selection does not establish safety of execution.

Where this fits with other work

Glean's JEV evaluation helped motivate this experiment. On 40 cases with positive expert transfers, it reported a median per-entry speedup of 8.1× over its existing LLM path. Results across its other tasks were mixed: JEV did not replace Glean’s production reranker. The speedup is supporting context from a different task and a small latency sample.

The closest comparison I found is Lyzr's Banking77 evaluation: it reports 80.1% zero-shot JEV accuracy on 770 messages, close to our 80.9%. It also reports 87.5% for an embedding classifier with ten labelled examples per intent and 93.9% for logistic regression trained on the full labelled set. Those aren't directly interchangeable measurements, but they expose an important gap in this project: we did not test a trained-classifier or embedding baseline.

This study therefore compares four implementations, rather than identifying the best available Banking77 classifier. A trained baseline is a useful next experiment, alongside adding new actions and measuring regressions in the original catalogue.

What I would test next: learning from corrections

The next question is whether reviewed mistakes can improve the selector without rebuilding the backend. My hypothesis is that a small domain-specific model, or better context for JEV, could help with recurring confusions. These are proposed experiments, not measured improvements in this study.

I would begin with supervised learning from corrected labels. Calling that reinforcement learning would be premature: we already have a direct target—the intended action. Learning a routing policy from delayed outcome rewards would be a separate experiment, with a harder problem of defining and attributing success.

There are three paths worth comparing:

Improve the information JEV receives. Clarify overlapping action descriptions and boundary cases, or retrieve a few relevant, human-reviewed examples into the request. TypeSafe’s customization documentation says customer fine-tuning and LoRA adapters are not supported; customization happens through state, instructions, criteria, and decomposed questions. Retrieved examples would make this a new, example-assisted configuration, not the zero-shot setup reported above.
Train a small local classifier. Start with embeddings plus logistic regression, trained on representative banking messages and reviewed corrections. Include ordinary successful cases as well as failures, so the training distribution is not dominated by exceptional cases. Evaluate this model on its own before deciding it belongs in a fallback.
Test a learned fallback or combining model. Use validation data to learn when the local classifier should override JEV, using signals such as disagreement, probability margins, or recurring intent confusions. A low-confidence trigger alone will miss confidently wrong selections. Another candidate is a model trained on probabilities from several narrow JEV questions, an approach also described in TypeSafe’s documentation. Unresolved cases would still abstain.

The evaluation matters as much as the mechanism. I would log both errors corrected and previously correct decisions broken, then compare overall accuracy, accuracy at equal referral volume, latency, cost, and incorrect selections of state-changing handlers. Better calibration alone would not demonstrate better classification or safe execution.

Corrections and threshold selection must use development data. The existing test set has already informed this analysis; if its failures guide changes, it becomes a regression set rather than an untouched final test. A new locked evaluation set—ideally from a later, representative workload—would be needed to support a fresh improvement claim. Any learned combining model should also use out-of-fold development predictions to avoid training on unrealistically easy, in-sample outputs.

Inspect the evidence

The switchless repository contains the TypeScript application, decision log, rules development history, tests, and all 16 original prediction files with SHA-256 hashes.

With Node 22+ and pnpm, the analysis can be regenerated without model API calls:

git clone https://github.com/grinder81/switchless.git
cd switchless
pnpm install
pnpm analyze

The generated analysis includes confidence intervals, paired comparisons, referral tie sensitivity, and the state-changing-handler breakdown. The raw-run notes distinguish runtime records from reconstructed values and project-log provenance. Original token counts were not retained; the cost values were calculated at runtime.

How I worked with AI

I framed and directed the experiment, chose the dataset and implementation constraints, required the product and a rules baseline, and reviewed the phase PRs. Claude wrote most of the implementation, rules, and measurement code. A subsequent review challenged the claims and led to preserved raw evidence, reproducible statistical analysis, and narrower conclusions. This article was also prepared with AI assistance and is being reviewed by me.

My practical conclusion is that a decision model can be a useful replacement for a hand-written language selector. In this experiment it improved accuracy and reduced local selection code, while the surrounding application remained essential. Choosing whether to execute a selected action needs its own evidence and controls.

If you maintain a similar dispatcher, what would determine your choice: labelled-data availability, latency, the cost of a wrong selection, or the effort of adding new actions?

DEV Community: Mo Al Mamun (Mamun)