DEV Community: Mo Al Mamun (Mamun)

Before attributing multilingual quantization loss, validate the likelihood path

Mo Al Mamun (Mamun) — Sun, 25 Oct 2026 15:13:26 +0000

My long-term goal is to make capable multilingual models practical on personal devices while preserving their usefulness across languages. I want to measure what compression changes, identify which parts of a model need more precision, and test whether allocating that precision differently can preserve quality within the same deployment size budget. Bangla is the language I want to investigate most deeply, with comparisons across other languages to keep the conclusions honest. The eventual test is both linguistic and practical: what can people do with the model, and what does it cost to run on a real device?

The starting point is personal. As a Canadian, I feel a particular connection to Cohere and want to explore what can be built with its multilingual models. This is independent work, not a claim of Cohere affiliation or endorsement. As a Bangla speaker, I also want this work to address a language I use and care about. Choosing Cohere's Tiny Aya and making Bangla the focus brings those interests together. That motivation determines where I start; it does not determine what the results should say.

The path through this article is: the model architecture, three runtime failures, probability validation, position sensitivity, fresh paired baselines, historical comparisons, and the limits of this checkpoint.

1. Establish what the evaluation measures

This article documents an early step toward that larger goal. Before testing a precision-allocation strategy, I needed to establish what the evaluation was actually measuring. The contribution here is a validated measurement procedure and fresh baselines. A sensitivity map, controlled precision-allocation experiments, broader language evaluation and real-device measurements remain later stages, each requiring its own evidence.

A multilingual quantization experiment can produce plausible accuracy numbers while measuring more than compression. The tokenizer, detokenizer, prompt format, request history and answer selection rule all sit between the model weights and the final table. A difference at any of those boundaries can look like a model capability difference.

This project began with a deployment question: how does compression affect Tiny Aya across languages, and could a later precision allocation recover quality at a fixed size budget? Before making that intervention claim, I needed a credible measurement path. That work encountered three distinct failures in the pinned local stack, then led to a fresh paired baseline. The result is an engineering checkpoint: a narrower, auditable comparison, with uncertainty and historical differences retained.

The model is CohereLabs/tiny-aya-global, a 3.35B Cohere2 model at revision af89d219b53ed9b13b8a4645f9c8028973510324. The comparison here uses a BF16 GGUF control and local Q4_K_M built from that pinned BF16 artifact without an importance matrix. It is not a survey of four-bit methods. The original BF16 vLLM reference remains a separately identified experiment.

The task is original-order multiple-choice reading comprehension from the Belebele benchmark (facebook/belebele, revision 7899cdfa4e1e0d733fd77c848e2c273cb1d32be2, CC-BY-SA 4.0) in English, Bangla and Hindi. The model is released under CC-BY-NC 4.0 for non-commercial use. The final fresh comparison uses the same 300 development questions per language in both arms. No held-out evaluation, human semantic annotation, generation experiment or phone-performance result is included.

2. The architecture defines candidate precision tests

The pinned model inspection records 36 decoder layers, hidden width 2,048, MLP width 11,008, 16 query heads and 4 key/value heads with head dimension 128, and a vocabulary of 262,144. Three sliding-attention layers with a 4,096-token window followed by one full-attention layer repeat nine times; the configured maximum is 8,192 positions. These are architecture facts, not measured sensitivity results. Source: pinned model inspection.

Each block normalizes the residual input once, sends that normalized input through attention and a gated SiLU MLP in parallel, then adds both outputs to the residual. After the decoder stack, a final LayerNorm feeds the output head, which shares weights with the token embedding. The Cohere2 implementation makes this parallel computation explicit.

The full model path is shown above a close-up of one decoder block. Teal attention and violet gated MLP branches share normalized input and join the original residual. Amber marks future precision tests; sensitivity and recovery remain unmeasured.

Candidate target	Hypothesis to test	Controlled test
Shared embedding/output head	Its precision may affect lexical representation and output scores.	Change the shared tensor allocation while accounting for tied weights and shipped bytes.
Attention Q/K/V/O projections	Attention projections may differ in sensitivity.	Compare projection groups under the same size budget.
MLP gate/up/down projections	Gating and expansion/reduction projections may differ in sensitivity.	Compare projection groups with real quantizer rounding.
Layer depth and local versus full attention	Sensitivity may vary by depth or attention type.	Compare depth/type allocations against uniform and random allocations.

These are hypotheses for recovering quality toward the BF16 comparator within a fixed size budget; they do not promise improvement above BF16. Allocation tests need matched shipped bytes, uniform and random controls, and actual quantizer rounding rather than nominal bit arithmetic alone. Accuracy and changed decisions must be assessed in English alongside Bangla and Hindi. Latency and memory effects require device profiling. KV-cache changes are a distinct runtime optimization and are not validated by these weight-precision tests.

3. Three failures, with different exposure

Native tokenization: changed input IDs

The first failure was in native tokenization. A minimal mismatch involved two spaces followed by 46: the pinned native Tiny Aya path did not preserve the reference digit splitting. Prefix-isolation and word-boundary/join controls distinguished the tokenizer behavior from an accidental interaction with surrounding text. In the selected parity probe, 5 of 150 contexts differed and only 580 of 600 complete candidate inputs matched; the patched path matched 600 of 600. These are scoped probe counts, not population prevalence. Expanded checking then found a join-control boundary case; the final correction matched 900 contexts and 3,600 candidate sequences, preserving the earlier patch as a distinct identity.

This is a token-identity issue: it changes what reaches the model before likelihood scoring starts. It should not be collapsed into the later Python decoding failure or used to label every historical result invalid. The pinned tokenizer investigation retains the controls and affected artifacts.

Python detokenization: a truncated round trip

The second failure was in the Python wrapper's detokenizer. A fixed 32-byte buffer was too small for some Bangla token pieces. The native API signaled the required larger buffer with a negative return value, but the wrapper did not correctly resize and retry. A reproduced prompt shrank from 899 to 851 characters on the broken round trip. The failed pilot stopped after completing 24 of 40 candidates; that is a progress count, not 24 affected cases. These are diagnostic and failed-pilot observations, not an estimate of population prevalence or evidence that every completed baseline contained that corruption.

The repair respects the native API contract: allocate the requested capacity, retry, and fail explicitly if decoding still fails. This was a local reproduction, not a claim of novel discovery: an earlier upstream report and proposed fix already existed (issue 2362 and PR 2363). My later PR 2388 remains a separate, overlapping proposal; coordination is pending. The recorded buffer investigation preserves the local reproducer and repair.

Byte-exact round-trip checks matter here. A visually plausible prompt, a successful HTTP response and a finite likelihood are insufficient evidence that the text survived the path unchanged.

Prefix reuse: request history changed likelihoods

The third failure emerged later, after strings and token counts were already matching. Echoed likelihoods depended on reused-prefix execution history. In the failed A3 validation, independent cold forwards disagreed with supplied-prompt log probabilities by as much as 0.18238 nats; tested chat continuation differences reached 0.02854 nats. A fresh-server request permutation also changed likelihoods. That made request order part of the measurement.

This dependence was observed in the llama-cpp-python server path used for scoring; it was not reproduced against native llama-server, so it is routed to the wrapper integration first rather than claimed as a core llama.cpp defect.

Instead of relaxing the numerical tolerance, I preserved the failed output and changed the diagnostic. Seven same-order wrapper completions were checked against an independent full-vocabulary float64 log-softmax of the raw logits. They matched the preserved HTTP arrays within 9.3e-7 nats. This bounded result helped separate execution-history effects from a parser or echo-alignment error. It was not a complete correctness gate.

The patched llama-cpp-python scoring wrapper subsequently forces cold prefill for echoed likelihood requests and bypasses cache restoration for that branch. It does not establish that caching is generally unsafe or that all nonscoring generation paths have been validated. A separate exact top-one ranking optimization removed a CPU bottleneck while preserving the tested ranking and tie behavior; its microbenchmark is not model throughput.

These failures have different exposure histories. A tokenizer discrepancy changes input IDs; a failed detokenization round trip corrupts the text interface; prefix reuse changes numerical execution for otherwise matching requests. The preserved evidence distinguishes diagnostic reproducers, failed pilots, historical completed runs and the fresh validated identities. That separation is necessary before claiming that a particular defect caused an accuracy change.

4. Validate probabilities before collecting another accuracy table

The final A3 runtime gate covered 192 candidates in 48 contexts per precision arm, followed by 36 candidates in a separate reversed-order check. The contexts covered all three languages, scoring variants and rotations, with longest-input and continuation cases. The independent reference used raw logits, rather than trusting the same HTTP response that the client parsed.

Validation required exact token IDs, one automatic BOS, matching echo strings and counts, and the expected single generated suffix. Numerical tolerance never excused token misalignment. Both the 0.0002-nat per-token limit and the summed continuation limit, max(0.001, continuation_tokens × 0.0002) nats, applied. The parser had its own 1e-8 agreement check against the supplied echo slice. Both arms passed, with no correctness failures or winner inversions in that coverage.

The later T1 baseline had its own, smaller T1-specific gate: 12 fixed contexts and all four candidates per arm (48 candidates), covering indices 0, 100, 299 and the longest input in each language, plus the same 48 candidates in reversed request order. All candidate continuations were independently checked to be single model tokens. These are bounded correctness checks, not a proof about arbitrary future prompts or implementations.

The evaluation path stayed on the supported upstream configurable task and likelihood scorer. I did not introduce a custom scoring adapter to make the probabilities agree. Runtime sources, native library, tokenizer, model bytes, commands and gate artifacts were hash-pinned. Resumption revalidated saved shards against the same identity rather than silently mixing environments.

5. Position sensitivity survives the runtime control

A2 had already shown substantial option-position sensitivity. Its historical runtime identity remains attached to those observations; the cache finding is an additional possible confound, not a retrospective explanation of every A2 flip.

A3 repeated the format question under the cold runtime. It used 100 questions per language, four cyclic option orders, and three variants: plain prompt with letter completion, tokenizer chat template with letter completion, and plain prompt with full option-text completion. Rotations and variants are repeated measurements of the same questions, not extra independent samples.

Under plain-letter scoring, all-four content consistency was 67%/65% in English, 43%/44% in Bangla and 52%/46% in Hindi for BF16/Q4. Consistency means selecting the same original answer content in all four orders, whether correct or wrong. Position sensitivity therefore remained in both precisions after the runtime control. It cannot be described as a quantization-only failure.

The same cold-runtime A3 plain-letter cells show the displayed-letter imbalance:

Language	BF16 predicted A/B/C/D	Q4 predicted A/B/C/D
English	141/89/90/80	147/87/85/81
Bangla	194/70/72/64	180/77/71/72
Hindi	169/69/83/79	185/65/75/75

Each cell contains 400 predictions across four rotations, with exactly 100 gold answers in each displayed position. These are A3 rotation histograms; T1's original-order gold distribution is different. The counts describe position sensitivity, not an internal defect or a validated correction.

The formats also changed measured performance. A3's complete results show exploratory chat-letter Q4−BF16 rotation-mean accuracy differences of −3.50 percentage points in English, with a 95% interval of [−5.95, −1.25], and −3.00 points in Bangla [−5.93, −0.25]. Those intervals exclude zero. All raw English-relative compression interaction intervals included zero, however. The evidence does not establish a Bangla-specific mechanism, and the many exploratory endpoints were not multiplicity adjusted.

Full option-text completion performed substantially worse here. Dividing its summed likelihood by displayed-choice Unicode character count changed results again. That normalization excludes the added ASCII-space delimiter; it is neither token nor byte normalization. For letter choices, normalization changes nothing because every displayed choice has one character. None of these diagnostics justified choosing an endpoint after seeing which one made compression look best.

The accepted downstream contract remained plain-letter summed continuation likelihood: one ASCII-space delimiter, no appended EOS, one automatic BOS, and the first displayed maximum on an exact tie. The next requirement was fresh full development comparators under that contract, not relabeling the 100-question rotation diagnostic as a 300-question baseline.

6. Fresh T1: small net differences, appreciable churn

Both fresh arms completed 180 five-question shards: 900 original-order questions and 3,600 candidate scores per arm. Every shard passed complete-membership, request, target, finite-score and provenance validation. The conditional overlap with A3 covered 300 original-order requests per arm; all 2,400 candidate comparisons across both arms matched without failures or winner inversions. Those rows were freshly collected, not copied from A3.

Language	BF16 GGUF correct /300	Q4_K_M correct /300	Q4−BF16, pp [95% interval]	Delta minus English, pp [95% interval]
English	230 (76.67%)	229 (76.33%)	−0.33 [−3.03, +2.36]	Reference
Bangla	187 (62.33%)	190 (63.33%)	+1.00 [−2.01, +4.15]	+1.33 [−2.63, +5.37]
Hindi	184 (61.33%)	180 (60.00%)	−1.33 [−4.70, +2.03]	−1.00 [−4.93, +2.97]

The intervals use 10,000 jointly resampled source-cluster draws, with 250 source clusters and seed 42. The same source draw is used across languages and paired arms. Questions sharing a source are not treated as independent, and the bootstrap denominator follows the sampled question count. The English-relative endpoint tests a difference in compression deltas, rather than confusing lower absolute Bangla or Hindi accuracy with extra compression damage.

All three fresh accuracy-difference intervals and both nontrivial English-relative intervals include zero. That is uncertainty, not equivalence or proof of no harm. The BF16 GGUF control already scores lower in Bangla and Hindi than English. This is a pre-quantization accuracy gap in this evaluation, not a comparison against a separate “base” model or evidence identifying its cause.

Net accuracy also hides changed decisions:

Language	Changed choices	Right→wrong	Wrong→right	Wrong→wrong
English	21/300 (7.00%)	9	8	4
Bangla	33/300 (11.00%)	10	13	10
Hindi	34/300 (11.33%)	15	11	8

Bangla churn is about 1.57 times English churn here, not doubled.

A small net delta can coexist with both regressions and improvements, and two wrong answers can still differ.

These counts motivate inspecting decisions; they do not by themselves identify a language-specific internal defect.

Original-order T1, 300 questions per language; BF16 GGUF control, not the separate vLLM reference. Intervals jointly resample 250 source clusters.

7. Historical results are preserved, not silently repaired

The fresh run was joined to historical A1 by exact source/question identity, targets and request/candidate bytes, with matching model artifacts. Within-arm historical-to-fresh changed predictions were 0/1/0 for BF16 and 7/12/10 for Q4 in English/Bangla/Hindi. Fresh Q4 correct counts rose from 226/189/176 to 229/190/180; BF16 moved from 230/188/184 to 230/187/184.

The score changes were larger than the earlier selected bridge might suggest. Across 1,200 paired candidate scores per language, maximum absolute Q4 changes were 0.89890 nats in English, 1.33790 in Bangla and 0.79041 in Hindi. The earlier 15-context bridge maximum was 0.42425 nats. Its smaller coverage never supplied a uniform bound on future score changes or accuracy.

These are combined observed runtime/settings transitions. They do not isolate a patch's causal effect. Keeping old results under their original identities, and publishing the score and margin distributions beside the fresh runs, is more informative than replacing the historical table with a corrected-looking number.

Historical and fresh identities remain distinct. These observed changes combine recorded runtime/settings differences; they are not isolated patch effects or universal error bounds.

8. Margins suggest a next analysis, not an explanation

The fresh BF16 top-two summed-score margin provides an exploratory view of the flips. English had 38 questions below one nat, with 15 flips; Bangla had 82, with 28 flips; Hindi had 83, with 29 flips. Above or equal to one nat, flips still occurred: 6/262, 5/218 and 5/217 respectively.

Lower-margin cases had more flips descriptively. The binning existed before this collection, but these counts have no source-cluster intervals or language-adjusted causal analysis. They do not show that margin explains away language, localize a damaged tensor, or select a precision repair. They establish a question for later controlled work.

9. What this checkpoint supports

The useful deliverable is a reproducible measurement contract and a fresh paired comparator, not a claim that multilingual compression is solved. The T1 evidence package preserves the complete report, raw shards, identities and validation evidence. Its report SHA-256 is c1ffee6b6030b1d5beb6c6afe3d855c1e1703df3e2fd6329270b06d55a1941a1. Independent internal checks reconstructed all 1,800 rows and recomputed the fresh source-bootstrap endpoints. That is an audit of this evidence, not external peer review.

Future work still needs human Bangla labels before the gated sweep, real-build and matched-byte controls before allocation claims, and a separate generation protocol. The recorded EOS 6 versus older EOS 3 discrepancy remains unresolved for generation; no-EOS likelihood candidates do not ratify a generation setting. Device latency, memory and energy require device measurements. The protected holdout was not used for selection or scoring.

For engineers building the same kind of evaluation, the transferable practice is concrete: verify bytes and token boundaries, compare parsed probabilities to independent logits, perturb request order, retain failures, and create new identities when the measurement path changes. Only then does the compression delta have a sufficiently clear meaning to guide the next experiment.

Sources and reproducibility: runtime validation and preserved failed probes, A2 historical rotations, A3 cold-runtime format results, accepted scoring contract, and fresh T1 report and reproducible archives. Dataset: the Belebele dataset card, revision 7899cdfa4e1e0d733fd77c848e2c273cb1d32be2, CC-BY-SA 4.0.

Can a Decision Model Replace Hand-Written Routing Rules? A 3,080-Message Experiment

Mo Al Mamun (Mamun) — Sat, 17 Oct 2026 16:45:02 +0000

A customer writes, “My card still hasn't arrived.” Somewhere in the backend, that sentence has to become a concrete choice: track an existing delivery, explain delivery times, or order another card.

I wanted to test whether a decision model could replace the hand-written logic that makes that choice.

So I built switchless: a fictional banking backend with 77 actions, four interchangeable dispatchers, and an evaluation on the 3,080-message Banking77 test set.

The result was useful, but less tidy than the original hypothesis. JEV beat our rules and answered about nine times faster than the LLM. The LLM was cheaper. At the same number of referrals, combining rules and JEV produced fewer incorrect automatic selections in an exploratory comparison, but still left roughly one wrong choice in eleven.

Disclosure: This is an independent experiment. I have no relationship with TypeSafe and received no credits, funding, or other support from them for this work.

The part of the backend I wanted to replace

This experiment is about interpreting language into an action. It isn't about replacing a gateway's endpoint table, authentication checks, permissions, or load balancing.

For an explicit request such as POST /cards/123/freeze, the destination is already known. For “I can't find my card,” the application has to interpret the request first.

The fictional product, Kestrel Bank, has 11 service modules and 77 implemented handlers. Some return information; others change an in-memory bank's state. They run inside one application, rather than as independently deployed microservices.

Customer message
      |
      v
POST /help
      |
      v
Interchangeable dispatcher
      |
      +--> Selected action --> Existing handler
      |
      +--> Abstain --> Escalation response / suggestions

Keeping the handlers and gateway shared let us change the selector without rebuilding the rest of the application.

Four ways to make the same choice

Rules: an ordered keyword-and-regex dispatcher, developed in eight stages. It abstains when no rule matches.

JEV: TypeSafe's jev-1.13.0, asked to choose among the 77 action IDs and their one-line descriptions.

LLM: OpenAI gpt-6-luna, given the same catalogue and customer message, with structured output and reasoning disabled.

Hybrid: run both rules and JEV. Accept their selection when they agree. When they disagree, accept JEV only if its confidence is at least 0.9; otherwise abstain.

The two model routers received no example messages in their requests. Here, “zero-shot” describes that setup; it doesn't establish whether either provider's training data included this public benchmark.

The rules were written by an AI coding assistant and reviewed as part of the project. That matters: this is a comparison against one disclosed implementation, not the best rule system a specialist team could build.

How the experiment was measured

Banking77 contains short banking-support messages labelled with 77 intents. Each intent maps to one action in Kestrel Bank.

We reserved 1,000 training messages as a held-out measurement sample and used the other 9,003 for rule development. A different 1,000-message sample, drawn from the rule-development pool, was used to choose the hybrid strategy and threshold. It was disjoint from the held-out measurement sample, but overlapped the data available for rule development. The final test evaluation used all 3,080 test messages. Rules, JEV, and hybrid settings were frozen before it; the LLM was added afterward in a separate run without retuning those systems.

The main metric was exact-action accuracy. Sending a message to the right service but choosing the wrong handler did not count as correct. Abstentions count as misses in overall accuracy and are reported separately.

The results

Measure	Rules	JEV	LLM	Hybrid
Exact-action accuracy	65.5%	80.9%	81.5%	74.1%
Referred / abstained	17.0%	0%	0%	18.3%
Accuracy on answered messages	78.9%	80.9%	81.5%	90.7%
Median decision time	0.29 ms	111 ms	975 ms	112 ms
API cost per 1,000	$0	$0.083	$0.028	$0.083
Router implementation lines	772	60	95	888

These are recorded-run measurements. API cost excludes infrastructure and human work. The hybrid's line count includes both underlying routers and its combining logic; the model integrations don't include provider-side model complexity.

JEV outperformed this rules implementation in each of the 11 service modules.

More rules helped, until the gains became small

On the 1,000-message held-out sample, the first 137 lines reached 39.4% accuracy. At 380 lines, accuracy was 55.2%. The final 772-line version reached 63.0%.

Later stages often repaired earlier rules. A concrete example: a pattern for cheque|check also caught the verb “check,” confusing a request about checking card status with a cheque-related intent.

The append-only development policy made the history inspectable, but it also inflated the accumulated code compared with normal maintenance, where an engineer could edit or remove an old rule. The curve describes this development process, not a universal ceiling for rules.

The effort curve uses the separate 1,000-message held-out sample. Lines measure local implementation size, not engineering hours or provider-side model complexity.

JEV was faster, while the LLM was cheaper

JEV's median decision took 111 ms, compared with 975 ms for Luna: about an 8.8-fold difference in these runs.

Accuracy was close. The LLM's observed advantage was 0.65 percentage points, with a paired 95% confidence interval from −0.38 to +1.68 points. No difference was detected on this test split; that does not prove equivalence. On the separate 1,000-message held-out sample, the LLM led by 2.20 points, with an interval from +0.19 to +4.21.

The price result contradicted the original expectation. The LLM cost about a third as much. Its repeated action catalogue benefited from cached-input pricing. Cheap classification isn't automatically cheaper than a low-cost generative model under a particular workload.

JEV was pinned to jev-1.13.0; the LLM had no pinned snapshot and was run once per evaluation split. On a fixed 200-message test subset, two repeat JEV runs changed 2.0% and 0.5% of selections relative to the original run; reversing the option order changed 5.0%.

The intervals describe uncertainty over messages. They don't cover model drift, different network conditions, or variation across repeated API runs.

Latency uses a logarithmic scale. The observed JEV–LLM accuracy difference was within the paired 95% confidence interval.

What the hybrid actually buys

The hybrid answered 2,516 messages and referred 564. Of the answered messages, 233 were incorrect: 90.7% accuracy at 81.7% coverage.

To put that beside the single-model routers, the analysis also referred each model's 564 least-confident messages. Confidence ties were broken by dataset index, with a sensitivity calculation for other tie orders.

Policy	Referrals	Accuracy on answered (95% CI)	Incorrect automatic selections
Hybrid	564	90.7% (89.5–91.8)	233
JEV, confidence-gated	564	88.8% (87.5–90.0)	282
LLM, confidence-gated	564	87.7% (86.3–88.9)	310

Intervals are 95% Wilson intervals for each policy’s answered accuracy, conditional on the selected messages; they do not account for selecting cutoffs on the test set. They are not intervals for the differences between policies.

This is an exploratory comparison at equal referral volume. The model cutoffs were selected on the test set to match the count, while the hybrid threshold was fixed beforehand. These policies refer different messages, which may require different amounts of human effort. Reordering boundary ties moved the LLM's answered accuracy between 87.3% and 88.1%; it did not reverse the observed ordering.

The hybrid made fewer incorrect automatic selections in this comparison; this observed difference does not establish superiority on future traffic. It also retained all the rules and called JEV on every message. That is a tradeoff, not a free improvement.

JEV’s threshold sweep is exploratory and uses the test split. Referrals are abstentions; human resolution and actual review effort were not measured.

Correct selection and safe execution are different questions

The gateway executes its selected handler immediately. Twenty handlers can change bank state, including blocking a card, opening a claim, or cancelling a pending transfer.

Of the hybrid's 233 incorrect selections, 32 selected a handler capable of changing state. These are not 32 observed harmful actions: whether a handler changes anything depends on the customer's account state, and the evaluation logs record predictions rather than real customer outcomes.

Likewise, an abstention only produces an escalation response or suggestions. We did not measure a human successfully resolving it.

This distinction changed how I describe the project. The results support replacing some hand-written intent-selection logic. They do not establish that the resulting actions are safe to execute without further checks or confirmation.

Higher confidence corresponded to higher accuracy, but did not guarantee correctness. Bins include at least 20 messages; calibration of intent selection does not establish safety of execution.

Where this fits with other work

Glean's JEV evaluation helped motivate this experiment. On 40 cases with positive expert transfers, it reported a median per-entry speedup of 8.1× over its existing LLM path. Results across its other tasks were mixed: JEV did not replace Glean’s production reranker. The speedup is supporting context from a different task and a small latency sample.

The closest comparison I found is Lyzr's Banking77 evaluation: it reports 80.1% zero-shot JEV accuracy on 770 messages, close to our 80.9%. It also reports 87.5% for an embedding classifier with ten labelled examples per intent and 93.9% for logistic regression trained on the full labelled set. Those aren't directly interchangeable measurements, but they expose an important gap in this project: we did not test a trained-classifier or embedding baseline.

This study therefore compares four implementations, rather than identifying the best available Banking77 classifier. A trained baseline is a useful next experiment, alongside adding new actions and measuring regressions in the original catalogue.

What I would test next: learning from corrections

The next question is whether reviewed mistakes can improve the selector without rebuilding the backend. My hypothesis is that a small domain-specific model, or better context for JEV, could help with recurring confusions. These are proposed experiments, not measured improvements in this study.

I would begin with supervised learning from corrected labels. Calling that reinforcement learning would be premature: we already have a direct target—the intended action. Learning a routing policy from delayed outcome rewards would be a separate experiment, with a harder problem of defining and attributing success.

There are three paths worth comparing:

Improve the information JEV receives. Clarify overlapping action descriptions and boundary cases, or retrieve a few relevant, human-reviewed examples into the request. TypeSafe’s customization documentation says customer fine-tuning and LoRA adapters are not supported; customization happens through state, instructions, criteria, and decomposed questions. Retrieved examples would make this a new, example-assisted configuration, not the zero-shot setup reported above.
Train a small local classifier. Start with embeddings plus logistic regression, trained on representative banking messages and reviewed corrections. Include ordinary successful cases as well as failures, so the training distribution is not dominated by exceptional cases. Evaluate this model on its own before deciding it belongs in a fallback.
Test a learned fallback or combining model. Use validation data to learn when the local classifier should override JEV, using signals such as disagreement, probability margins, or recurring intent confusions. A low-confidence trigger alone will miss confidently wrong selections. Another candidate is a model trained on probabilities from several narrow JEV questions, an approach also described in TypeSafe’s documentation. Unresolved cases would still abstain.

The evaluation matters as much as the mechanism. I would log both errors corrected and previously correct decisions broken, then compare overall accuracy, accuracy at equal referral volume, latency, cost, and incorrect selections of state-changing handlers. Better calibration alone would not demonstrate better classification or safe execution.

Corrections and threshold selection must use development data. The existing test set has already informed this analysis; if its failures guide changes, it becomes a regression set rather than an untouched final test. A new locked evaluation set—ideally from a later, representative workload—would be needed to support a fresh improvement claim. Any learned combining model should also use out-of-fold development predictions to avoid training on unrealistically easy, in-sample outputs.

Inspect the evidence

The switchless repository contains the TypeScript application, decision log, rules development history, tests, and all 16 original prediction files with SHA-256 hashes.

With Node 22+ and pnpm, the analysis can be regenerated without model API calls:

git clone https://github.com/grinder81/switchless.git
cd switchless
pnpm install
pnpm analyze

The generated analysis includes confidence intervals, paired comparisons, referral tie sensitivity, and the state-changing-handler breakdown. The raw-run notes distinguish runtime records from reconstructed values and project-log provenance. Original token counts were not retained; the cost values were calculated at runtime.

How I worked with AI

I framed and directed the experiment, chose the dataset and implementation constraints, required the product and a rules baseline, and reviewed the phase PRs. Claude wrote most of the implementation, rules, and measurement code. A subsequent review challenged the claims and led to preserved raw evidence, reproducible statistical analysis, and narrower conclusions. This article was also prepared with AI assistance and is being reviewed by me.

My practical conclusion is that a decision model can be a useful replacement for a hand-written language selector. In this experiment it improved accuracy and reduced local selection code, while the surrounding application remained essential. Choosing whether to execute a selected action needs its own evidence and controls.

If you maintain a similar dispatcher, what would determine your choice: labelled-data availability, latency, the cost of a wrong selection, or the effort of adding new actions?