Same 652 papers. Same six questions. Three different judges. No ground truth — so this is a story about agreement, not accuracy. Read it as such.
The setup
On 2026-11-13, the Paper Radar daily run fetched 652 new arXiv papers across cs.AI, cs.CL, cs.LG, cs.IR and stat.ML. Each paper was judged against three interests (agent evaluation, LLM calibration, RAG & retrieval — retrieval weighted 0.9), one exclusion (medical imaging), plus two signals: paper type (new method / benchmark / dataset / survey / theory / …) and whether code is released.
Three backends judged the identical set with the identical questions:
- TypeSafe Jev (jev-1.13.0) — judged 2026-11-13, 08:53 UTC
- Claude Haiku 5.5 — judged 2026-11-13, 19:00 UTC
- Gemini Decision 1 — judged 2026-11-20, on the same 2026-11-13 listing
The Gemini run used the new backend in PR #29 (stdlib-only, same Backend protocol as the Jev and Claude backends). It ran on the free tier — 652 calls, $0.
Thresholds are each model's own: Jev 0.80 / 0.50, Claude 0.65 / 0.35 (fitted on 49 labelled papers), Gemini 0.80 / 0.50 — the Jev defaults, uncalibrated. That last point matters; keep it in mind for every count below.
Results
| Jev | Claude Haiku 5.5 | Gemini Decision 1 | |
|---|---|---|---|
| Judged | 652 | 649 | 652 |
| Failed | 0 | 3 | 0 |
| Must-read | 56 | 41 | 47 |
| Maybe | 83 | 73 | 62 |
| Skip | 508 | 524 | 536 |
| Excluded | 5 | 11 | 7 |
| Input tokens | 612,233 | 681,175 | 524,861 |
| Cost (paid price) | $0.026 | $0.10 | $0.05 |
| Cost (actually paid) | $0.026 | $0.10 | $0 (free tier) |
| Judging time | 41 s | ~12 min | ~21 min* |
* Gemini's 21 minutes is free-tier pacing (50,000 tokens/min), not model speed. The Decisions API quotes 150–250 ms per call; at the paid tier's 1,000,000 tokens/min the same run would take a few minutes. Claude's 12 minutes is API latency across 652 calls; Jev's 41 seconds is a single batched model.
How much do they agree?
Mean |Δ relevance| is the average absolute difference in the combined relevance score (0–1). Lower means the two models score papers more alike.
| Pair | Must-read overlap | Shortlist overlap | Mean abs. diff. relevance | Paper-type agreement |
|---|---|---|---|---|
| Jev vs Claude | 30 of 41 / 56 | 98 of 114 / 139 | 0.157 | 85.8% |
| Jev vs Gemini | 36 of 47 / 56 | 100 of 109 / 139 | 0.111 | 92.6% |
| Claude vs Gemini | 33 of 41 / 47 | 94 of 109 / 114 | 0.082 | 81.2% |
Per interest, Claude and Gemini are closest on agent evaluation (mean |Δ| 0.037) and retrieval (0.028); calibration is the loosest for every pair (0.065–0.111).
The three-way picture: 28 papers are must-reads for all three models; 73 are must-read for at least one. Jev alone flags 18 papers the other two don't; Claude alone flags 6, Gemini alone 6.
Three things I take from this
1. The two general-purpose models cluster together; Jev is the outlier. Claude–Gemini relevance differs by 0.082 on average, roughly half the Jev–Claude gap (0.157). The two LLM backends see these papers more alike than either sees them like Jev. That is not a verdict on who is right — there are no labels here — but it is a consistent pattern across all three interests.
2. Gemini sits in the middle, and it is the most reliable of the three on this run. 47 must-reads (between Jev's 56 and Claude's 41), 7 excluded (between 5 and 11), zero failures in 652 calls. Claude failed 3 papers with malformed replies; Gemini's typed answers can't malform, so its only failure mode is the network. But its thresholds are uncalibrated, so don't over-read the counts: 47 is what the Jev defaults produce, not what a fitted Gemini threshold would.
3. Thresholds move the answer as much as the model does. Put Claude on Jev's 0.80 / 0.50 thresholds and its shortlist collapses from 114 papers to 68; its must-reads drop from 41 to 36. Any comparison of "how many must-reads" is really a comparison of thresholds. The probability agreement (mean |Δ|) is the sturdier number.
One more wrinkle: paper type is the noisiest signal. Jev and Gemini agree on it 92.6% of the time, but Claude and Gemini only 81.2% — the two models that agree most on relevance agree least on what kind of paper it is.
Honest limits
- No labels, no accuracy. Agreement is consensus, not correctness. Jev and Claude already have a screening benchmark (Cochrane review CD012347: both 100% recall, Jev 92.5% work saved vs Claude 90.3%). Gemini hasn't run it yet — that's the next step.
- Gemini's thresholds are uncalibrated. 0.80 / 0.50 are Jev's defaults. Fitted thresholds need labelled Gemini judgments.
- Different judging days. Jev and Claude judged on 2026-11-13; Gemini on 2026-11-20. Model versions may have shifted in between.
- Two papers were judged on their v1 abstracts (they were retitled after 2026-11-13); two more titles were normalised from a LaTeX escape the RSS feed stored. Negligible at n=652, but stated.
- Costs are paid-tier prices. The Gemini run itself cost $0 on the free tier; at paid prices it would be ~$0.05.
What's next
- Run Gemini on the CD012347 screening benchmark, the same way Jev and Claude were measured — that turns agreement into accuracy.
- Calibrate Gemini's thresholds on labelled papers, instead of borrowing Jev's.
- The Gemini backend itself is open for review in PR #29.
Paper Radar is open source (Eliot5566/JEV-Paper-Radar, MIT). The comparison scripts and data live in the workspace, not the repo — ask in the comments if you want the per-paper numbers.
Top comments (0)