Benchmarking Jev: what a decision model can (and can't) do in an agent harness

艾特玖 — Mon, 21 Sep 2026 08:23:48 +0000

Model under test: jev-1.13.0 · Date: Sep 2026 · Code & raw results: github.com/Aitejiu/jev-harness-lab

A black-box engineering evaluation: 10 public datasets, ~22,500 API calls, 52.2M input tokens, $2.19 total.

Jev is TypeSafe AI's "System One" decision model. You send it a state plus typed questions; it returns typed answers with probabilities and confidence. It never generates text — and it costs about $0.00004 and 0.3s per call. That makes it a candidate for the "fast-think layer" of an agent harness: the flood of narrow decisions an agent makes every turn.

I spent a few days measuring where that actually works.

TL;DR

Works (directly usable in a harness):

Capability	Dataset	Result
Indirect prompt-injection detection	InjecAgent (1,105)	at threshold 0.10: P/R 100%, 0% false positives
Reranking	BEIR SciFact (900 pairs)	BM25 → Jev: MRR 0.622 → 0.843, Hit@1 50% → 78.3%
Intent classification	SNIPS / Banking77	97.9% (7 classes) / 80.3% (77 classes)
Tool catalog routing	MetaTool (199 tools)	96.5% with similar distractors (k=5)
Skill routing	SkillRetBench (501 skills)	two-stage R@1 75.8% vs 38.0% best baseline
Shell command risk gate	130 hand-built commands	100% dangerous caught, 98.2% safe passed
Tool-relevance detection	BFCL (1,140)	77.0% live accuracy after iteration (59.5% first try)

Doesn't work (negative results):

Task	Result	Why
Model-difficulty routing	51% accuracy (no signal)	requires predicting another model's failure modes
Trajectory failure attribution	AUROC 0.560 (random)	requires cross-step causal reasoning
Non-English tasks	KO R@1 48.9% vs EN 61.5%	English-primary training

Two engineering lessons worth more than the benchmarks:

For fuzzy semantic judgments, orthogonal decomposition + code composition beats single-question prompting (shell-gate false positives 14.5% → 1.8%).
For multi-label decisions, let choice compete first, then let noul verify — multi-skill routing went from 9.0% to 81.0% R@1.

Model gives calibrated local judgments; code holds the control flow.

1. The model and its interface

POST https://api.typesafe.ai/v1/systemone
Authorization: Bearer $TYPESAFE_API_KEY

{
  "state": "...",                # string | object | array, text only
  "model": "jev-latest",
  "questions": { "..." : Question }
}

Primitive	Meaning	Returns
`noul`	yes/no	probability 0–1 (no separate confidence field)
`choice`	pick one of a closed set	choice + probability distribution + confidence
`score`	rate on an ordered rubric	probability-weighted score + legend + confidence

Known limits, all verified during testing:

choice caps at 255 options — beyond that, split into stages.
state + questions share ~32k tokens; large corpora must be retrieved first.
Text only; English-primary (CJK accuracy drops).
Pricing reference: $42 / billion input tokens.

2. How I evaluated

Every experiment follows the same shape (eval/ in the repo): load a dataset → build state/questions → concurrent calls with JSONL caching → threshold sweeps, coverage curves, calibration and cost.

Metric definitions matter, so being explicit:

Threshold sweep — treat the returned probability as a score; report precision/recall/F1/FPR at each threshold.
Confidence gating curve — noul has no confidence field, so certainty is proxied by max(p, 1-p); report coverage vs accuracy-within-coverage.
Conditional Hit@1 (SkillRet) — fraction of queries where the gold candidate was in the shortlist and Jev picked it; isolates the selector from retrieval.
Cost — input tokens × $42/B (output-token billing, if any, would add slightly).

Datasets: InjecAgent, neuralchemy prompt-injection, synthesized tool-output injections, BEIR SciFact, SNIPS, Banking77, MetaTool ToolE, BFCL v3, SkillRet, SkillRetBench, RouterBench, Who&When, plus a 130-command shell risk set.

3. Results

3.1 Indirect injection detection

Real data: 17 user tools × 62 attacker instructions (InjecAgent).

Threshold	Precision	Recall	F1	Benign FPR
0.50	100%	89.8%	94.7%	0%
0.10	100%	100%	100%	0%

By attack type (t=0.5): data-stealing 98.5% recall, direct-harm 80.6%; benign mean score 0.03.
Gating: certainty ≥ 0.90 auto-handles 41.1% of traffic at 100% accuracy.
Cost: $0.0213 for 1,105 calls; P50 0.30s.

Synthetic tool outputs (injections wrapped inside WebFetch/Read/GitHub/Email results): at 0.5, P 97.1 / R 85.0 / FPR 4.9%. Misses are encoding bypasses (Cyrillic homoglyphs, code-snippet disguises).

Cautionary dataset (neuralchemy, 942): the labels include "requests for harmful content," which my criteria did not ask about — recall drops to 55.1% at 0.5, purely a task-definition mismatch. Also note: the 0–0.1 score bucket still contains 16.6% malicious samples. Low score is not a safety guarantee.

3.2 Reranking (BEIR SciFact)

60 queries × 15 candidates; Jev scores each (query, document) pair 0–3, then reranks.

Ranker	Recall@5	MRR	nDCG@5	Hit@1
BM25	74.2%	0.622	0.632	50.0%
Jev	90.0%	0.843	0.848	78.3%

Mean score: 2.38 for relevant vs 0.65 for irrelevant. $0.028 / 900 pairs, P50 0.31s.

3.3 Intent and catalog routing

Dataset	Classes	Top-1	Gating
SNIPS (1,400)	7	97.9%	conf≥0.90: 93.6% coverage, 99.1% acc
Banking77 (3,080)	77	80.3%	conf≥0.90: 67.8% coverage, 92.8% acc
MetaTool (199 tools)	199	96.5% (k=5, similar distractors)	—

MetaTool comparison: the paper reports 69.1% for ChatGPT on the same "similar choices" subtask (different exact setup — treat as magnitude reference). Errors cluster on near-duplicate tools: descriptions need explicit not_for boundaries.

3.4 Shell command risk gate

Design: four orthogonal noul questions (destructive / touches secrets / exfiltrates / irreversible) + code that combines them into deny/review/allow.

Variant	3-class agreement	Dangerous caught	Safe passed
v1 (loose criteria)	66.2%	85.0%	87.3%
v2 (tightened criteria)	78.5%	100%	98.2%
Single-question choice baseline	77.7%	100%	85.5%

v1 missed availability harms (fork bombs, reboot, firewall lockouts) until the criteria explicitly included them. The decomposition's payoff shows up in false positives: 14.5% → 1.8%.

3.5 Tool-relevance detection (BFCL)

Task: given a request plus a function list, should any function be called? 1,140 samples.

Version	Method	classic acc	live-irrelevance acc	FPR
v1	single `noul` "is it relevant"	84.2%	59.5%	35.2%
v2	criteria tightened to "purpose match"	89.6%	67.3%	27.9%
v2c	composed: `purpose × (1 − incidental)`	90.8%	77.0%	20.1%

Typical failure mode: "technically possible" mistaken for "semantically intended" (a generic requests.get judged relevant).

3.6 Skill routing: 60% → 76%

Goal: stop injecting the entire skill catalog into the main model. Route each request with one fast judgment, load only the winner.

SkillRet (6,006 skills, 300 queries) — isolating selector from retrieval:

Setup	BM25 Hit@1	shortlist recall	end-to-end	conditional Hit@1*
k=10, name+description	52.7%	71.3%	63.7%	89.3%
k=20, + body in criteria	52.7%	77.3%	65.3%	84.5%
k=10, body in BM25 index	58.3%	78.7%	68.3%	86.9%

* conditional = gold in the shortlist. The selector is strong (85–89%); the bottleneck is retrieval. Adding bodies to the BM25 index helped retrieval; adding them to Jev's criteria did not.

SkillRetBench (501 skills, official baselines, macros over 5 settings):

Metric	BM25	NaiveLLM	Jev (chunked)	Jev (hybrid)
Recall@1	38.0%	30.2%	60.4%	75.8%
Recall@10	59.8%	55.6%	87.6%	93.0%
nDCG@10	53.4%	45.1%	60.3%	70.2%

The architecture iteration is the interesting part (multi-skill composition R@1):

Setting	v1 chunked (one pick per chunk)	v2 multiselect (per-candidate noul)	v3 hybrid
multi-skill composition	9.0%	15.0%	81.0%
overall macro R@1	60.4%	—	75.8%

v2 failed because per-candidate noul scores have no competition: everything clusters around 0.5 (top-10 mean 0.56) and the merged ranking is noise. v3 fixes it: chunked choice produces 15 candidates with real probability spread, then one request with 15 noul questions verifies set membership. Compete first, verify second.

Cost: hybrid $0.650 / 500 queries (P50 2.07s); the production-shaped "BM25 top-50 + Jev" is $0.072 / 500 queries at P50 0.33s (R@1 54.8%).

3.7 Negative results

Model-difficulty routing (RouterBench, 825 samples): routing decision accuracy 51.3% — no signal; scores skew toward "a small model can handle it" (median 0.11). Chinese subset 14.6% vs English 57.7%. Predicting which model will fail is not a natural-language property.

Trajectory failure attribution (Who&When, 184 trajectories × 8 steps): step-level AUROC 0.560, top-1 localization 16.3% (random 12.5%). The label requires "never corrected" — cross-step causality, outside a System One model's shape.

4. Versus published numbers

On identical candidate lists (SkillRouter paper, ~80K pool, top-20):

System	Hit@1
GPT-4o-mini (listwise judge)	67.3%
GPT-5.4-mini (listwise judge)	66.0%
Qwen3-Reranker-8B	71.4%
SkillRouter 1.2B (fine-tuned)	74.0%
R3 (fine-tuned, bilingual)	77.1%
Jev (zero-shot, conditional)	86.9–89.3%

The paper's own conclusion: LLM-as-judge baselines are "not competitive." Letting a general LLM rank directly loses to purpose-built rerankers. Jev, zero-shot and untrained, sits in the same band as fine-tuned rerankers under the conditional metric (different pool sizes and metrics — read as order-of-magnitude, not a controlled comparison).

One more finding from the SkillRet paper worth remembering: off-the-shelf rerankers can hurt when the first-stage retriever is strong (nDCG@10 dropped 83.5 → 74–78 there). If you bolt Jev onto a strong embedding retriever, re-measure the marginal gain.

5. Six engineering rules

Criteria are the decision boundary. Merge "pressure" and "legitimate escalation" into one question and you get a mushy 0.47; split them and you get 0.04 / 0.96.
Orthogonal decomposition + code composition beats one clever prompt. Shell gate false positives 14.5% → 1.8%; tool relevance 59.5% → 77.0%.
High scores are reliable; low scores are not safe. In injection data, everything ≥0.1 was 90%+ malicious — but the 0–0.1 bucket still contained 16.6% malicious. Keep a human-review band.
Confidence ≠ correctness. High confidence means the model is sure it applied your definition — including your mistakes (I saw wrong routings at confidence 1.0).
Multi-label: compete, then verify (section 3.6).
Respect the capability boundary. Only ask about locally observable patterns in text — not model capability (51%), not cross-step causality (AUROC 0.56).

6. Getting started

Two integrations ship with this work:

MCP server (three tools: scan_injection, bash_risk, rank_candidates):

uvx jev-mcp        # PyPI; or: uvx --from git+https://github.com/Aitejiu/jev-harness-lab jev-mcp

Agent skill (skill routing, load only the winner):

npx skills add Aitejiu/jev-harness-lab --skill jev-skill-router

Both need TYPESAFE_API_KEY.

7. Reproducing

git clone https://github.com/Aitejiu/jev-harness-lab
cd jev-harness-lab
uv venv --python 3.12 .venv
uv pip install -r requirements.txt
echo "TYPESAFE_API_KEY=<your-key>" > .env

.venv/bin/python eval/run_injecagent.py --concurrency 6      # injection
.venv/bin/python eval/run_rerank.py --queries 60            # reranking
.venv/bin/python eval/run_bash.py --variant v2              # shell gate
.venv/bin/python eval/run_skillretbench.py --variant hybrid --per-setting 100

All raw results are committed under eval/results/; add --report-only to regenerate reports from cache. Datasets are public and downloaded separately per eval/README.md.

8. Limitations

Single model version (jev-1.13.0); re-run after upgrades.
Some datasets are samples or hand-built (130 shell commands, 121 synthetic injections, 60 SciFact queries) — selection bias possible.
Thresholds are reference values; calibrate on your own data (use the coverage–accuracy curve).
Cost estimates are input-token only.
English-primary; CJE tasks need dedicated validation.
SkillRetBench's NaiveLLM/SADO baselines are simulated per the dataset authors.

If you build agent infrastructure and want to swap notes on decision models in the harness, the repo issues are open — or find me on X. The 22,500 calls cost $2.19; the lessons were cheaper than the tokens.

DEV Community: 艾特玖