JEV-style local judges in three spoonfuls: when voting improves a decision and when it only stalls it

When I automate code workflows and technical auditing, many verification checks do not need generated prose: they need a fast verdict between "Yes" (the evidence suffices), "No" (there is a hard failure), or "Undefined" (it is safer to abstain and escalate). Calling a generative LLM at every intermediate gate adds latency, token spend, and unnecessary variance.

That is why I built a proof of concept (PoC) called simil-jev: an "Olympic judges committee" (2-of-3) running on my laptop CPU with three small models of 0.4B to 0.8B parameters (Laya¹, VegaML², and Arc Decide³). Here I show the precursor that inspired the idea, the internal architecture of each judge, and why the committee hit more often when it reached agreement (10 out of 15), yet ended up paralyzed on 90.62% of the test bank.

---

Spoonful 1: The JEV precursor and the promise of an "Olympic judges committee"

The spark for this prototype came from JEV Ultrafast⁴, an experiment by the Browser Use team together with TypeSafe AI. Instead of asking a heavy LLM to inspect screenshots or write paragraphs at every browser step, jev-ultrafast turns the page into an indexed table of controls and delegates the choice to Jev, a specialized System 1 decision model that selects the atomic action (CLICK, SELECT, WAIT, BLOCKED) in milliseconds and only calls a small LLM when it needs to type text (TYPE_TEXT).

⚡ JEV Ultrafast (Browser Use × TypeSafe AI)

browser-use/jev-ultrafast
TypeSafe Jev Docs

Precursor in action: Zurich → London flight search on Google Flights solved in 7.1 s by separating atomic decision-making (Jev) from text generation (asset served from the official repository).

Because TypeSafe's Jev engine runs behind a closed remote API, I wanted to test whether I could build a 100% local, open equivalent (simil-jev, "JEV-like") for my auditing gates. And since no sub-billion-parameter model is infallible on its own, I combined them under an "Olympic judges committee" architecture (2-of-3 majority with the right to abstain):

• The promise (Condorcet 1785⁵; Dietterich 2000⁶): under Condorcet's jury theorem, if each judge is right more than half the time (p > 0.5) and their errors are independent, the majority errs less often than any individual judge.
• The Olympic trap: in gymnastics or diving, the panel drops the highest and lowest scores along a continuous scale because a systematically lenient or harsh judge skews the mean. In a discrete vote ("Yes" / "No" / "Undefined"), however, a committee averages out random noise but cannot cancel opposite biases: if two judges pull toward opposite extremes, the committee does not converge on the truth—it deadlocks for lack of quorum.

---

Spoonful 2: The three judges on the bench and the dual CPU bottleneck

To assemble the committee, I picked three open projects with radically different internal designs. Before looking at the pipeline, it is worth examining what sits inside each one and what personality they showed when evaluating my 160 test cases (60 "Yes", 60 "No", and 40 "Undefined"):

Judge 1 · The approver

Laya 0.4.11

GitHub
HF Model
Demo Space

Multilingual non-autoregressive System 1 decision engine trained with strictly proper scoring rules (RLCD) to emit single-pass verdicts without generating text.

CPU cost: 2.26 GiB RAM · 13.83 s cold · 366 ms p95
Panel behavior: voted "Yes" on 125/160 cases (including 48 of the 60 negatives → 73 false positives).

Judge 2 · The rejecter

VegaML 0.11.0

GitHub PyPI 0.11.0 Qwen3.5-0.8B

Couples a frozen reader (Qwen3.5-0.8B) with a compact 14.3M-parameter physics engine that projects states onto a landscape of potential wells and simulates a damped particle until it settles into an answer.

CPU cost: 4.93 GiB RAM · 13.66 s cold · 1,717 ms p95
Panel behavior: voted "No" on 131/160 cases and hit 0 out of 60 on the "Yes" class.

Judge 3 · The silent one

Arc Decide 0.4B

Arc53/arc-decide-0.4b ONNX Runtime

Compact 0.4B-parameter classifier exported to ONNX Runtime (INT8) to evaluate relevance, contracts, and binary gates in under 1 GiB of memory.

CPU cost: 0.59 GiB RAM · 4.21 s cold · 950 ms p95
Panel behavior: under its panel threshold (0.66) it stayed silent on 148/160 cases; it only voted 12 times (all "Yes", 0 "No" votes).

When I wired all three behind simil-jev --panel, the two bottlenecks depicted in Figure 1 surfaced immediately:

Figure 1. Real pipeline of simil-jev --panel. Stage 3 exposes the physical bottleneck (three sequential CPU subprocesses totaling ~31.7 s cold when reloading weights on every call) and Stage 4 exposes the logical bottleneck (Laya and VegaML clash on 101/160 cases while Arc stays silent).

On a workstation with a dedicated GPU and all three engines kept resident in VRAM, Stage 3 would take milliseconds. To understand why the exact opposite happened on my laptop, here is the hardware profile of the machine where I ran the experiment (ffpoke, my alias wrapping fastfetch + pokeget inside kitty and tmux):

Test machine (ffpoke in kitty + tmux). Lenovo YOGA 730-15IKB laptop with a 15 W Intel Core i7-8550U processor (4 physical cores / 8 Kaby Lake-R threads with AVX2), integrated Intel UHD 620 graphics without CUDA, and 23.19 GiB of DDR4 RAM on EndeavourOS.

On that i7-8550U, without resident daemons permanently locking 8 GiB of RAM, spawning three Python and PyTorch interpreters sequentially cost roughly 31.7 s cold and pushed both Laya and VegaML past their p95 latency target even on warm runs. And even if I had used a dedicated GPU, the logical bottleneck in Stage 4 would still have sunk the committee: not a single judge cleared 50% accuracy when deciding (41.41%, 40.80%, and 47.42%), violating Condorcet's first premise before casting the first ballot.

---

Spoonful 3: The jury ballots and the 66.67% paradox

To see why the committee hit 10 out of 15 when it reached agreement (66.67% selective accuracy) and still proved useless in practice, it suffices to inspect three real ballots from my 240-case bank (80 calibration and 160 test cases):

Family 1 · Contractual precedence (Why "No" vanished) "An unsigned draft does not replace the signed record. The signed record says red and the draft says green. Is the active value green?" → Ground truth: "No" Laya: YES ✗ VegaML: NO ✓ Arc: ABSTAIN (conf < 0.66) 2-of-3 Verdict: UNDEFINED (Deadlock) Family 2 · Direct evidence (When the tie-breaker certifies an error) "The expected hash is aa00 and the observed hash is bb11. Do the hashes match?" (evidence_direct-14-v3) → Ground truth: "No" Laya: YES ✗ VegaML: NO ✓ Arc: YES ✗ 2-of-3 Verdict: YES (False Positive) Family 3 · Ambiguity and insufficiency (Shared bias) "The date appears as 04/05, without year or regional format. Is it May 4, 2026?" (negation_ambiguity_insufficient-18-v2) → Ground truth: "Undefined" Laya: YES ✗ VegaML: YES ✗ Arc: ABSTAIN 2-of-3 Verdict: YES (False Positive)

Those three ballots sum up the entire experiment better than a spreadsheet of numbers:

1. "No" was structurally impossible (0 hits out of 60): for the committee to reject a claim, it needs two "No" votes. Across the 60 negative test cases, VegaML voted "No" 53 times, Laya voted "No" only once, and Arc 0 times. Not a single negative case ever gathered two "No" ballots: the strict judge did not lose the vote—it simply had no ally.
2. The 66.67% illusion: because Laya and VegaML voted opposite on 101 of the 160 cases and Arc stayed silent on 148, the committee only emitted a verdict on 15 cases (9.38% coverage). It got 10 of those 15 right (66.67%, with a 95% Wilson interval of 41.7% to 84.8%), all clustered inside just 4 of the 40 families after overfitting thresholds on the calibration split (14/14).
3. Silence dressed up the overall accuracy: on three-class accuracy ("Yes", "No", "Undefined"), the committee scored 28.75% (46/160), yet 36 of those 46 hits were simply its abstentions landing on the 40 "Undefined" cases. A one-line script that always answers "Undefined" already scores 25.00% without spending RAM or waiting half a minute:

Final scoreboard across the 160 test cases (60 "Yes" · 60 "No" · 40 "Undefined")

Contrast between accuracy when the evaluator dares to decide (selective accuracy + coverage) and overall three-class accuracy.

🏆 Arc Decide 0.4B (solo) — Best standalone model 0.59 GiB · 4.21 s cold · 49 FP · 5/60 on "No" Hit rate when deciding (coverage 60.63%)47.42% 3-class accuracy (68/160)42.50%
Laya 0.4.11 (solo) — The optimist 2.26 GiB · 13.83 s cold · 73 FP · 52/60 on "Yes" Hit rate when deciding (coverage 80.00%)41.41% 3-class accuracy (66/160)41.25%
VegaML 0.11.0 (solo) — The pessimist 4.93 GiB · 13.66 s cold · 4 FP · 51/60 on "No" Hit rate when deciding (coverage 78.13%)40.80% 3-class accuracy (61/160)38.13%
⚖️ 2-of-3 Committee (Laya + VegaML + Arc) — Deadlock by opposite biases ~31.7 s serial · 3 FP · 0/60 on "No" Hit rate when deciding (10/15 · coverage 9.38%)66.67% 3-class accuracy (36 of 46 are abstentions)28.75%
Trivial baseline: always answer "Undefined" 0.00 GiB · 0 ms · 0 FP Hit rate when deciding (coverage 0.00%)N/A 3-class accuracy (40/160 free hits)25.00%

---

Research takeaways: which open model could run in milliseconds on my i7-8550U

When would a JEV-style classifier committee actually make sense? When each judge clears 50% accuracy on every class out-of-sample (Condorcet 1785⁵), when their errors are uncorrelated (Dietterich 2000⁶), when their probabilities are properly calibrated (Guo et al. 2017⁷), and when at least two judges have real sensitivity to vote "No". Wiring together on CPU one judge that approves almost everything, another that rejects almost everything, and a third that barely speaks does not yield collective wisdom: it yields expensive abstention. That is why I retired the prototype.

Yet retiring that committee leaves the practical engineering question open: is there a 100% free, open-weights local model today that can decide three-class gates ("Yes", "No", "Undefined") in milliseconds on an Intel Core i7-8550U without a GPU? I believe so, provided I switch architectural families and stop treating logical verification as a text-generation problem; but that is a hypothesis this experiment did not test.

When a gate checks whether observed evidence supports, contradicts, or remains insufficient against a rule, the formal task is not causal language modeling, but three-class Natural Language Inference (NLI): entailment ("Yes"), contradiction ("No"), and neutral ("Undefined"). In that category, the candidate I propose to measure next is cross-encoder/nli-deberta-v3-small, built on the DeBERTaV3 architecture (He, Gao, and Chen 2023⁸); its repository also ships an ONNX INT8 build for AVX2 CPUs that runs with onnxruntime, and optimum can export others. The idea of reusing an NLI model as a classifier comes from Laurer et al. (2024⁹), whose model family includes a larger variant, MoritzLaurer/DeBERTa-v3-base-mnli-fever-anli. I have not yet measured either one on my 160-case bank:

Hypothesis to measure on CPU: nli-deberta-v3-small (ONNX INT8)

HF Model (Apache) DeBERTaV3 Paper ONNX Runtime

Bidirectional cross-encoder with disentangled attention (separating content and relative position) pre-trained via replaced token detection (ELECTRA) and fine-tuned on SNLI + MultiNLI. It evaluates premise and hypothesis jointly in a single forward pass without loading PyTorch.

Latency still to be measured
I did not run it on this bank. Because it decides in a single pass without generating text, my hypothesis is that it will land well below the committee's 366–1,717 ms warm and ~31.7 s cold.

~165 MiB on disk (ONNX INT8 for AVX2)
44M backbone parameters (142M total with vocabulary embeddings); the model_quint8_avx2.onnx file takes about 165 MiB on disk. Its runtime RAM is still to be measured, so I do not yet compare it with the roughly 7.8 GiB of RAM used by the three judges.

3 native classes ("Yes" / "No" / "Undefined")
Emits direct logits for contradiction, entailment, and neutral (87.55% on MNLI mismatched, according to its model card). Whether that fixes the "No" blindness (0/60) on my gates remains to be measured.

The takeaway I keep from this proof of concept has a measured part and an unmeasured one. Measured: on a work laptop without a dedicated accelerator, a committee of mismatched models did not beat its best individual judge on three-class accuracy and never managed to say "No". Still a hypothesis: that a single small NLI cross-encoder, calibrated and compiled to ONNX INT8, would beat it in latency, memory footprint, and sensitivity to say "No". When verification is deterministic (a hash, a JSON schema, or an exit code), a 0 ms code rule remains unbeatable; and when I genuinely need to judge semantic entailment or contradiction locally, my next step is not to stack three judges in series, but to measure an architecture designed for that task first.

---

References

---

1. NandhaKishorM (ConvAI Innovations). 2026a. "Laya: System 1 Non-Autoregressive Multilingual Decision Engine". GitHub repository. https://github.com/NandhaKishorM/laya. ↩

2. NandhaKishorM. 2026b. "VegaML: Potential-Well Physics Decision Engine over Frozen Language Backbones". GitHub repository. https://github.com/NandhaKishorM/vegaml. ↩

3. Arc53. 2025. "arc-decide-0.4b-v0.2: Compact ONNX Decision Classifier". Hugging Face model repository. https://huggingface.co/Arc53/arc-decide-0.4b-v0.2. ↩

4. Browser Use and TypeSafe AI. 2026. "JEV Ultrafast: Sub-Second Browser Agent Actions with Jev Decision Models". Official GitHub repository. https://github.com/browser-use/jev-ultrafast. ↩

5. Condorcet, Jean-Antoine-Nicolas de Caritat, Marquis de. 1785. Essai sur l'application de l'analyse à la probabilité des décisions rendues à la pluralité des voix. Paris: Imprimerie Royale. ↩

6. Dietterich, Thomas G. 2000. "Ensemble Methods in Machine Learning". In Multiple Classifier Systems (MCS 2000), Lecture Notes in Computer Science, vol. 1857, 1–15. Berlin: Springer. Scholar ↩

7. Guo, Chuan, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. "On Calibration of Modern Neural Networks". In Proceedings of the 34th International Conference on Machine Learning (ICML), PMLR 70: 1321–1330. ↩

8. He, Pengcheng, Jianfeng Gao, and Weizhu Chen. 2023. "DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing". In International Conference on Learning Representations (ICLR 2023). arXiv · Scholar ↩

9. Laurer, Moritz, Wouter van Atteveldt, Andreu Casas, and Kasper Welbers. 2024. "Less Annotating, More Classifying: Addressing the Data Scarcity Issue of Supervised Machine Learning with Deep Transfer Learning and BERT-NLI". Political Analysis 32 (1): 84–100. Model from the same family: https://huggingface.co/MoritzLaurer/DeBERTa-v3-base-mnli-fever-anli. DOI · Scholar ↩