DEV Community

Eliot5566
Eliot5566

Posted on Edited on

Three decision models, one arXiv morning: Jev vs Claude Haiku vs Gemini Decision 1 on 652 papers

Same 652 papers. Same six questions. Three different judges. No ground truth — so this is a story about agreement, not accuracy. Read it as such.

The setup

On 2026-11-13, the Paper Radar daily run fetched 652 new arXiv papers across cs.AI, cs.CL, cs.LG, cs.IR and stat.ML. Each paper was judged against three interests (agent evaluation, LLM calibration, RAG & retrieval — retrieval weighted 0.9), one exclusion (medical imaging), plus two signals: paper type (new method / benchmark / dataset / survey / theory / …) and whether code is released.

Three backends judged the identical set with the identical questions:

  • TypeSafe Jev (jev-1.13.0) — judged 2026-11-13, 08:53 UTC
  • Claude Haiku 5.5 — judged 2026-11-13, 19:00 UTC
  • Gemini Decision 1 — judged 2026-11-20, on the same 2026-11-13 listing

The Gemini run used the new backend in PR #29 (stdlib-only, same Backend protocol as the Jev and Claude backends). It ran on the free tier — 652 calls, $0.

Thresholds are each model's own: Jev 0.80 / 0.50, Claude 0.65 / 0.35 (fitted on 49 labelled papers), Gemini 0.80 / 0.50 — the Jev defaults, uncalibrated. That last point matters; keep it in mind for every count below.

Results

Jev Claude Haiku 5.5 Gemini Decision 1
Judged 652 649 652
Failed 0 3 0
Must-read 56 41 47
Maybe 83 73 62
Skip 508 524 536
Excluded 5 11 7
Input tokens 612,233 681,175 524,861
Cost (paid price) $0.026 $0.10 $0.05
Cost (actually paid) $0.026 $0.10 $0 (free tier)
Judging time 41 s ~12 min ~21 min*

* Gemini's 21 minutes is free-tier pacing (50,000 tokens/min), not model speed. The Decisions API quotes 150–250 ms per call; at the paid tier's 1,000,000 tokens/min the same run would take a few minutes. Claude's 12 minutes is API latency across 652 calls; Jev's 41 seconds is a single batched model.

How much do they agree?

Mean |Δ relevance| is the average absolute difference in the combined relevance score (0–1). Lower means the two models score papers more alike.

Pair Must-read overlap Shortlist overlap Mean abs. diff. relevance Paper-type agreement
Jev vs Claude 30 of 41 / 56 98 of 114 / 139 0.157 85.8%
Jev vs Gemini 36 of 47 / 56 100 of 109 / 139 0.111 92.6%
Claude vs Gemini 33 of 41 / 47 94 of 109 / 114 0.082 81.2%

Per interest, Claude and Gemini are closest on agent evaluation (mean |Δ| 0.037) and retrieval (0.028); calibration is the loosest for every pair (0.065–0.111).

The three-way picture: 28 papers are must-reads for all three models; 73 are must-read for at least one. Jev alone flags 18 papers the other two don't; Claude alone flags 6, Gemini alone 6.

Three things I take from this

1. The two general-purpose models cluster together; Jev is the outlier. Claude–Gemini relevance differs by 0.082 on average, roughly half the Jev–Claude gap (0.157). The two LLM backends see these papers more alike than either sees them like Jev. That is not a verdict on who is right — there are no labels here — but it is a consistent pattern across all three interests.

2. Gemini sits in the middle, and it is the most reliable of the three on this run. 47 must-reads (between Jev's 56 and Claude's 41), 7 excluded (between 5 and 11), zero failures in 652 calls. Claude failed 3 papers with malformed replies; Gemini's typed answers can't malform, so its only failure mode is the network. But its thresholds are uncalibrated, so don't over-read the counts: 47 is what the Jev defaults produce, not what a fitted Gemini threshold would.

3. Thresholds move the answer as much as the model does. Put Claude on Jev's 0.80 / 0.50 thresholds and its shortlist collapses from 114 papers to 68; its must-reads drop from 41 to 36. Any comparison of "how many must-reads" is really a comparison of thresholds. The probability agreement (mean |Δ|) is the sturdier number.

One more wrinkle: paper type is the noisiest signal. Jev and Gemini agree on it 92.6% of the time, but Claude and Gemini only 81.2% — the two models that agree most on relevance agree least on what kind of paper it is.

Honest limits

  • No labels, no accuracy. Agreement is consensus, not correctness. Jev and Claude already have a screening benchmark (Cochrane review CD012347: both 100% recall, Jev 92.5% work saved vs Claude 90.3%). Gemini hasn't run it yet — that's the next step.
  • Gemini's thresholds are uncalibrated. 0.80 / 0.50 are Jev's defaults. Fitted thresholds need labelled Gemini judgments.
  • Different judging days. Jev and Claude judged on 2026-11-13; Gemini on 2026-11-20. Model versions may have shifted in between.
  • Two papers were judged on their v1 abstracts (they were retitled after 2026-11-13); two more titles were normalised from a LaTeX escape the RSS feed stored. Negligible at n=652, but stated.
  • Costs are paid-tier prices. The Gemini run itself cost $0 on the free tier; at paid prices it would be ~$0.05.

What's next

  • Run Gemini on the CD012347 screening benchmark, the same way Jev and Claude were measured — that turns agreement into accuracy.
  • Calibrate Gemini's thresholds on labelled papers, instead of borrowing Jev's.
  • The Gemini backend itself is open for review in PR #29.

Paper Radar is open source (Eliot5566/JEV-Paper-Radar, MIT). The comparison scripts and data live in the workspace, not the repo — ask in the comments if you want the per-paper numbers.

Top comments (0)