Judge-only intervals miss the full-human value 18 of 21 times. PPI++ saves ~9% of human labels; disagreement routing costs 2.6x — pre-registered and coverage-gated.
Scope note, first paragraph as promised: 7 English-heavy annotation tasks, 3 mid-tier judges (2 local, 1 API), one run each, $0.29 API spend. Numbers below are scoped to the pre-registered study (846 configs × 2,000 replicates).
Your LLM judge says the system scores 72%. The true number — what full human
labeling would conclude — is 65%. And the ±1% interval you computed describes
the judge's opinions, not the truth. I measured exactly this: judge-only
intervals missed the full-human value in 18 of 21 task-and-panel
combinations across 7 real annotation tasks.
The usual fix — "label a random 100 by hand to check" — is ad hoc. Nobody
knows whether 100 is enough, which 100 to pick, or how to combine them with
the 5,000 judge labels. FewHumans is a library that answers all three
with valid statistics: it routes which items humans label, combines the two
sources with prediction-powered inference, and tells you when to stop looking
at anytime-valid intervals.
Setup
Three judges (two local Qwen 9B/4B through Ollama, one gpt-4o-mini — $0.29
total) labeled 7 tasks: HelpSteer2 and SummEval ratings, MT-Bench and Arena
preferences, WMT translation scores, toxicity flags, and OracleBench math
verdicts. Prompts frozen before any simulation. Ground truth = the human
labels; strategies only "see" a label when they buy it.
Results
Pre-registered, 846 configurations × 2,000 replicates each:
- PPI++: 0.91 [0.87, 0.95] — saves ~9% of human labels. Modest, real.
- Stratified PPI++: 0.91 [0.84, 0.98] — same savings, simpler assumptions.
- Fixed PPI: 2.27× — costs labels. With mediocre judges (agreement 45–77%), untuned λ=1 adds noise. Never run PPI without power tuning.
- Disagreement routing: 2.64× — costs labels on 6 of 7 tasks, even though disagreement predicts errors (AUROC up to 0.80). Inverse-probability weights blow up variance faster than the signal helps. My prior (from Sfyraki & Wang) said this might fail. It failed.
- Judge-only: misses 18/21. Verbalized confidence is overconfident everywhere (calibration error up to 0.98).
Every default interval passed a 196-configuration coverage gate first
(Student-t intervals; stratified needs n≥100; skewed truth needs n≥200).
Limitations
Savings are ~9%, not 10× — garbage in applies. 7 English-heavy tasks, 3
mid-tier judges, one run each. Ratio estimands (F1) not shipped. Active
routing needs logged probabilities some workflows lack. High-stakes domains:
this reduces human review, never removes it.
Reproduce
pip install fewhumans
pytest # 47 tests
python examples/toy_end_to_end.py
Repo: github.com/raihan-js/fewhumans · Dataset: JudgeBank (8 tasks, every
judge output + cost + latency) · Paper draft: arXiv soon.
Top comments (1)
The disagreement-routing result is the part I'd want more people to sit with. It's such an intuitive move — "humans should look at the cases where the judge is unsure" — and the AUROC up to 0.80 makes it look like it should work. But inverse-probability weights blowing up variance faster than the signal helps is the trap nobody warns you about until you've already shipped the dashboard. We hit a milder version of this routing human review toward model-disagreement cases and watched our confidence intervals get wider, not tighter, before we understood why.
The thing I keep coming back to is that "label a random 100 by hand" isn't just ad hoc — it's ad hoc in a way that feels rigorous, which is worse. You get a number, it has a ± on it, and the ± is describing the judge's opinions rather than the truth. Quick question: for the PPI++ ~9% savings, did the gain hold up as judge agreement improved, or is the whole method basically a correction for mediocre judges that stops mattering once your judge is actually good?