Model under test:
jev-1.13.0· Date: Sep 2026 · Code & raw results: github.com/Aitejiu/jev-harness-labA black-box engineering evaluation: 10 public datasets, ~22,500 API calls, 52.2M input tokens, $2.19 total.
Jev is TypeSafe AI's "System One" decision model. You send it a state plus typed questions; it returns typed answers with probabilities and confidence. It never generates text — and it costs about $0.00004 and 0.3s per call. That makes it a candidate for the "fast-think layer" of an agent harness: the flood of narrow decisions an agent makes every turn.
I spent a few days measuring where that actually works.
TL;DR
Works (directly usable in a harness):
| Capability | Dataset | Result |
|---|---|---|
| Indirect prompt-injection detection | InjecAgent (1,105) | at threshold 0.10: P/R 100%, 0% false positives |
| Reranking | BEIR SciFact (900 pairs) | BM25 → Jev: MRR 0.622 → 0.843, Hit@1 50% → 78.3% |
| Intent classification | SNIPS / Banking77 | 97.9% (7 classes) / 80.3% (77 classes) |
| Tool catalog routing | MetaTool (199 tools) | 96.5% with similar distractors (k=5) |
| Skill routing | SkillRetBench (501 skills) | two-stage R@1 75.8% vs 38.0% best baseline |
| Shell command risk gate | 130 hand-built commands | 100% dangerous caught, 98.2% safe passed |
| Tool-relevance detection | BFCL (1,140) | 77.0% live accuracy after iteration (59.5% first try) |
Doesn't work (negative results):
| Task | Result | Why |
|---|---|---|
| Model-difficulty routing | 51% accuracy (no signal) | requires predicting another model's failure modes |
| Trajectory failure attribution | AUROC 0.560 (random) | requires cross-step causal reasoning |
| Non-English tasks | KO R@1 48.9% vs EN 61.5% | English-primary training |
Two engineering lessons worth more than the benchmarks:
- For fuzzy semantic judgments, orthogonal decomposition + code composition beats single-question prompting (shell-gate false positives 14.5% → 1.8%).
- For multi-label decisions, let
choicecompete first, then letnoulverify — multi-skill routing went from 9.0% to 81.0% R@1.
Model gives calibrated local judgments; code holds the control flow.
1. The model and its interface
POST https://api.typesafe.ai/v1/systemone
Authorization: Bearer $TYPESAFE_API_KEY
{
"state": "...", # string | object | array, text only
"model": "jev-latest",
"questions": { "..." : Question }
}
| Primitive | Meaning | Returns |
|---|---|---|
noul |
yes/no | probability 0–1 (no separate confidence field) |
choice |
pick one of a closed set | choice + probability distribution + confidence |
score |
rate on an ordered rubric | probability-weighted score + legend + confidence |
Known limits, all verified during testing:
-
choicecaps at 255 options — beyond that, split into stages. -
state+questionsshare ~32k tokens; large corpora must be retrieved first. - Text only; English-primary (CJK accuracy drops).
- Pricing reference: $42 / billion input tokens.
2. How I evaluated
Every experiment follows the same shape (eval/ in the repo): load a dataset → build state/questions → concurrent calls with JSONL caching → threshold sweeps, coverage curves, calibration and cost.
Metric definitions matter, so being explicit:
- Threshold sweep — treat the returned probability as a score; report precision/recall/F1/FPR at each threshold.
-
Confidence gating curve —
noulhas no confidence field, so certainty is proxied bymax(p, 1-p); report coverage vs accuracy-within-coverage. - Conditional Hit@1 (SkillRet) — fraction of queries where the gold candidate was in the shortlist and Jev picked it; isolates the selector from retrieval.
- Cost — input tokens × $42/B (output-token billing, if any, would add slightly).
Datasets: InjecAgent, neuralchemy prompt-injection, synthesized tool-output injections, BEIR SciFact, SNIPS, Banking77, MetaTool ToolE, BFCL v3, SkillRet, SkillRetBench, RouterBench, Who&When, plus a 130-command shell risk set.
3. Results
3.1 Indirect injection detection
Real data: 17 user tools × 62 attacker instructions (InjecAgent).
| Threshold | Precision | Recall | F1 | Benign FPR |
|---|---|---|---|---|
| 0.50 | 100% | 89.8% | 94.7% | 0% |
| 0.10 | 100% | 100% | 100% | 0% |
- By attack type (t=0.5): data-stealing 98.5% recall, direct-harm 80.6%; benign mean score 0.03.
- Gating: certainty ≥ 0.90 auto-handles 41.1% of traffic at 100% accuracy.
- Cost: $0.0213 for 1,105 calls; P50 0.30s.
Synthetic tool outputs (injections wrapped inside WebFetch/Read/GitHub/Email results): at 0.5, P 97.1 / R 85.0 / FPR 4.9%. Misses are encoding bypasses (Cyrillic homoglyphs, code-snippet disguises).
Cautionary dataset (neuralchemy, 942): the labels include "requests for harmful content," which my criteria did not ask about — recall drops to 55.1% at 0.5, purely a task-definition mismatch. Also note: the 0–0.1 score bucket still contains 16.6% malicious samples. Low score is not a safety guarantee.
3.2 Reranking (BEIR SciFact)
60 queries × 15 candidates; Jev scores each (query, document) pair 0–3, then reranks.
| Ranker | Recall@5 | MRR | nDCG@5 | Hit@1 |
|---|---|---|---|---|
| BM25 | 74.2% | 0.622 | 0.632 | 50.0% |
| Jev | 90.0% | 0.843 | 0.848 | 78.3% |
Mean score: 2.38 for relevant vs 0.65 for irrelevant. $0.028 / 900 pairs, P50 0.31s.
3.3 Intent and catalog routing
| Dataset | Classes | Top-1 | Gating |
|---|---|---|---|
| SNIPS (1,400) | 7 | 97.9% | conf≥0.90: 93.6% coverage, 99.1% acc |
| Banking77 (3,080) | 77 | 80.3% | conf≥0.90: 67.8% coverage, 92.8% acc |
| MetaTool (199 tools) | 199 | 96.5% (k=5, similar distractors) | — |
MetaTool comparison: the paper reports 69.1% for ChatGPT on the same "similar choices" subtask (different exact setup — treat as magnitude reference). Errors cluster on near-duplicate tools: descriptions need explicit not_for boundaries.
3.4 Shell command risk gate
Design: four orthogonal noul questions (destructive / touches secrets / exfiltrates / irreversible) + code that combines them into deny/review/allow.
| Variant | 3-class agreement | Dangerous caught | Safe passed |
|---|---|---|---|
| v1 (loose criteria) | 66.2% | 85.0% | 87.3% |
| v2 (tightened criteria) | 78.5% | 100% | 98.2% |
| Single-question choice baseline | 77.7% | 100% | 85.5% |
v1 missed availability harms (fork bombs, reboot, firewall lockouts) until the criteria explicitly included them. The decomposition's payoff shows up in false positives: 14.5% → 1.8%.
3.5 Tool-relevance detection (BFCL)
Task: given a request plus a function list, should any function be called? 1,140 samples.
| Version | Method | classic acc | live-irrelevance acc | FPR |
|---|---|---|---|---|
| v1 | single noul "is it relevant" |
84.2% | 59.5% | 35.2% |
| v2 | criteria tightened to "purpose match" | 89.6% | 67.3% | 27.9% |
| v2c | composed: purpose × (1 − incidental)
|
90.8% | 77.0% | 20.1% |
Typical failure mode: "technically possible" mistaken for "semantically intended" (a generic requests.get judged relevant).
3.6 Skill routing: 60% → 76%
Goal: stop injecting the entire skill catalog into the main model. Route each request with one fast judgment, load only the winner.
SkillRet (6,006 skills, 300 queries) — isolating selector from retrieval:
| Setup | BM25 Hit@1 | shortlist recall | end-to-end | conditional Hit@1* |
|---|---|---|---|---|
| k=10, name+description | 52.7% | 71.3% | 63.7% | 89.3% |
| k=20, + body in criteria | 52.7% | 77.3% | 65.3% | 84.5% |
| k=10, body in BM25 index | 58.3% | 78.7% | 68.3% | 86.9% |
* conditional = gold in the shortlist. The selector is strong (85–89%); the bottleneck is retrieval. Adding bodies to the BM25 index helped retrieval; adding them to Jev's criteria did not.
SkillRetBench (501 skills, official baselines, macros over 5 settings):
| Metric | BM25 | NaiveLLM | Jev (chunked) | Jev (hybrid) |
|---|---|---|---|---|
| Recall@1 | 38.0% | 30.2% | 60.4% | 75.8% |
| Recall@10 | 59.8% | 55.6% | 87.6% | 93.0% |
| nDCG@10 | 53.4% | 45.1% | 60.3% | 70.2% |
The architecture iteration is the interesting part (multi-skill composition R@1):
| Setting | v1 chunked (one pick per chunk) | v2 multiselect (per-candidate noul) | v3 hybrid |
|---|---|---|---|
| multi-skill composition | 9.0% | 15.0% | 81.0% |
| overall macro R@1 | 60.4% | — | 75.8% |
v2 failed because per-candidate noul scores have no competition: everything clusters around 0.5 (top-10 mean 0.56) and the merged ranking is noise. v3 fixes it: chunked choice produces 15 candidates with real probability spread, then one request with 15 noul questions verifies set membership. Compete first, verify second.
Cost: hybrid $0.650 / 500 queries (P50 2.07s); the production-shaped "BM25 top-50 + Jev" is $0.072 / 500 queries at P50 0.33s (R@1 54.8%).
3.7 Negative results
Model-difficulty routing (RouterBench, 825 samples): routing decision accuracy 51.3% — no signal; scores skew toward "a small model can handle it" (median 0.11). Chinese subset 14.6% vs English 57.7%. Predicting which model will fail is not a natural-language property.
Trajectory failure attribution (Who&When, 184 trajectories × 8 steps): step-level AUROC 0.560, top-1 localization 16.3% (random 12.5%). The label requires "never corrected" — cross-step causality, outside a System One model's shape.
4. Versus published numbers
On identical candidate lists (SkillRouter paper, ~80K pool, top-20):
| System | Hit@1 |
|---|---|
| GPT-4o-mini (listwise judge) | 67.3% |
| GPT-5.4-mini (listwise judge) | 66.0% |
| Qwen3-Reranker-8B | 71.4% |
| SkillRouter 1.2B (fine-tuned) | 74.0% |
| R3 (fine-tuned, bilingual) | 77.1% |
| Jev (zero-shot, conditional) | 86.9–89.3% |
The paper's own conclusion: LLM-as-judge baselines are "not competitive." Letting a general LLM rank directly loses to purpose-built rerankers. Jev, zero-shot and untrained, sits in the same band as fine-tuned rerankers under the conditional metric (different pool sizes and metrics — read as order-of-magnitude, not a controlled comparison).
One more finding from the SkillRet paper worth remembering: off-the-shelf rerankers can hurt when the first-stage retriever is strong (nDCG@10 dropped 83.5 → 74–78 there). If you bolt Jev onto a strong embedding retriever, re-measure the marginal gain.
5. Six engineering rules
- Criteria are the decision boundary. Merge "pressure" and "legitimate escalation" into one question and you get a mushy 0.47; split them and you get 0.04 / 0.96.
- Orthogonal decomposition + code composition beats one clever prompt. Shell gate false positives 14.5% → 1.8%; tool relevance 59.5% → 77.0%.
- High scores are reliable; low scores are not safe. In injection data, everything ≥0.1 was 90%+ malicious — but the 0–0.1 bucket still contained 16.6% malicious. Keep a human-review band.
- Confidence ≠ correctness. High confidence means the model is sure it applied your definition — including your mistakes (I saw wrong routings at confidence 1.0).
- Multi-label: compete, then verify (section 3.6).
- Respect the capability boundary. Only ask about locally observable patterns in text — not model capability (51%), not cross-step causality (AUROC 0.56).
6. Getting started
Two integrations ship with this work:
MCP server (three tools: scan_injection, bash_risk, rank_candidates):
uvx jev-mcp # PyPI; or: uvx --from git+https://github.com/Aitejiu/jev-harness-lab jev-mcp
Agent skill (skill routing, load only the winner):
npx skills add Aitejiu/jev-harness-lab --skill jev-skill-router
Both need TYPESAFE_API_KEY.
7. Reproducing
git clone https://github.com/Aitejiu/jev-harness-lab
cd jev-harness-lab
uv venv --python 3.12 .venv
uv pip install -r requirements.txt
echo "TYPESAFE_API_KEY=<your-key>" > .env
.venv/bin/python eval/run_injecagent.py --concurrency 6 # injection
.venv/bin/python eval/run_rerank.py --queries 60 # reranking
.venv/bin/python eval/run_bash.py --variant v2 # shell gate
.venv/bin/python eval/run_skillretbench.py --variant hybrid --per-setting 100
All raw results are committed under eval/results/; add --report-only to regenerate reports from cache. Datasets are public and downloaded separately per eval/README.md.
8. Limitations
- Single model version (
jev-1.13.0); re-run after upgrades. - Some datasets are samples or hand-built (130 shell commands, 121 synthetic injections, 60 SciFact queries) — selection bias possible.
- Thresholds are reference values; calibrate on your own data (use the coverage–accuracy curve).
- Cost estimates are input-token only.
- English-primary; CJE tasks need dedicated validation.
- SkillRetBench's NaiveLLM/SADO baselines are simulated per the dataset authors.
If you build agent infrastructure and want to swap notes on decision models in the harness, the repo issues are open — or find me on X. The 22,500 calls cost $2.19; the lessons were cheaper than the tokens.
Top comments (0)