I expected the neural entailment model to win. It didn't.
I wanted to know how far the cheap, no-API-call hallucination checks get on real RAG output before you need an LLM judge or a human. So I built three small detectors, ran them on RAGTruth, and compared them. The one that costs microseconds per response beat the one that costs about seven seconds on a CPU.
The setup
RAGTruth (Niu et al., ACL 2024, MIT licence) has about 18,000 responses from six LLMs to retrieval-grounded prompts, in three tasks: question answering, summarisation and data-to-text. Humans annotated hallucinated spans. I call a response hallucinated if it has at least one annotated span.
Each detector takes a source and a response and returns a score in [0, 1]:
- lexical: the share of content tokens in the response (lowercased, stopwords and punctuation removed, numbers kept) that never appear in the source.
-
numeric: the share of numbers in the response not found in the source, after normalising
1,000/1000/1k. -
nli: SummaC-style. Each response sentence is checked with
cross-encoder/nli-deberta-v3-smallagainst its three closest source sentences (by token overlap) and their concatenation. The response scores as its least-supported sentence. - ensemble: logistic regression on the three scores.
Thresholds and ensemble weights were fit on a stratified train sample of 299 responses and then frozen. Everything below is on a stratified test subsample of 900 of the 2,700 test responses (300 per task), with a 35.3% hallucination base rate.
The result
| detector | AUROC (95% CI) | precision | recall | F1 |
|---|---|---|---|---|
| lexical | 0.778 (0.746-0.806) | 0.476 | 0.858 | 0.612 |
| nli | 0.716 (0.681-0.747) | 0.476 | 0.836 | 0.607 |
| numeric | 0.535 (0.503-0.566) | 0.353 | 1.000 | 0.522 |
| ensemble | 0.766 (0.737-0.797) | 0.450 | 0.934 | 0.607 |
The lexical and numeric detectors are cheap enough to also run on all 2,700 test responses. Lexical gets AUROC 0.785 there, so the subsample isn't flattering it. The intervals for lexical and NLI barely overlap on the subsample, but I'd treat the gap as "lexical is at least as good here", not as a law of nature.
The ensemble, fit on 299 responses, didn't beat lexical alone overall. It only came out ahead on QA.
Why the cheap one wins (my read, not proven)
Most RAGTruth hallucinations add content: a name, a claim, a plausible-sounding detail the source never mentions. A check for "new vocabulary" catches a lot of that for free. The NLI model has a harder job, and my setup gives it a hard time in places: it only sees the top three source sentences, and its probabilities are saturated near 0 and 1 (the tuned threshold ended up at 0.9987).
On data-to-text, where the source is flattened JSON, NLI is at chance:
| task | hallucinated | lexical | nli | ensemble |
|---|---|---|---|---|
| QA | 20% | 0.708 | 0.706 | 0.740 |
| Summary | 20% | 0.616 | 0.627 | 0.582 |
| Data2txt | 66% | 0.661 | 0.496 | 0.663 |
AUROC within each task, 300 responses each. Note how much lower the per-task numbers are than the pooled 0.78: part of the pooled score comes from the detectors picking up on the task itself, since base rates differ so much between tasks.
The per-model picture is mixed too. NLI is the best detector on Mistral 7B instruct (0.812 vs 0.688 for lexical) and the worst on GPT-4 (0.689 vs 0.808). With 150 responses per model the intervals are wide, so read the pattern, not the third decimal.
The two kinds of detector also fail differently. In examples.json there are cases where NLI flags a clean-looking fabrication ("will make it the longest F1 circuit ever built", lexical score 0.18, NLI 0.9997) and cases where NLI lets plausible boilerplate through that the lexical check catches. That's the argument for combining them, and the ensemble result above is the argument against assuming it just works.
The caveats, because there are several
Precision is poor. At the threshold tuned for F1, the lexical detector catches 85.8% of hallucinated responses with precision 0.476. Working that through with the 35.3% base rate, about half of the correct responses get flagged too (this is my arithmetic from the table, not a number the benchmark reports directly). A flag is roughly a coin flip. Predicting "hallucinated" for everything gives F1 0.522, so an F1 of 0.61 is a modest gain, and AUROC is the fairer yardstick.
Thresholds are tuned on 299 responses to maximise F1. They won't transfer to your data. The numeric detector's best threshold was 0.0, which flags everything with a number missing from the source, so its row is just the all-positive baseline.
NLI ran on a subsample. It takes about 7.3 seconds per response on a loaded 4-thread CPU, so the full run took 2 h 33 min and I used 900 test responses instead of 2,700.
I did not run an LLM judge. The interface is in the repo, but there are no results for it, so none of this says anything about how these cheap checks compare with an LLM-as-judge. I'd expect the judge to be more accurate and also more expensive, but that's an expectation, not a finding.
Scope. English only, three tasks, 2023-era generator models, and response-level labels (span positions are ignored). I also didn't fine-tune anything on RAGTruth, which would flatter the numbers on this benchmark.
What I'd do with it
Treat a cheap detector as a triage signal, not a gate. If you block or rewrite every flagged answer, you'll degrade a lot of good ones. If you use the score to decide which answers a person reads first, it's useful. And the ranking between detectors changed with task and model, so the only way to know what works on your traffic is to label a few hundred of your own responses and measure.
Reproduce it
Code: github.com/protforge-ai/groundcheck
git clone https://github.com/protforge-ai/groundcheck && cd groundcheck
pip install -e ".[test]"
python -m pytest
groundcheck bench --seed 0
bench downloads the RAGTruth files, scores the train and test samples, and writes metrics.json, by_task.csv, by_model.csv, test_scores.csv and examples.json into results/. NLI scores are cached, so an interrupted run resumes. To score one response against its source:
groundcheck score --source source.txt --response response.txt --detector nli
--detector is one of lexical, numeric, nli. It prints JSON with a response score and a per-sentence verdict. The default --threshold of 0.5 is not tuned, so tune it on your own data.
The code is MIT licensed, built by ProtForge. The full write-up, with a per-model table, is at protforge.com/research/hallucination-detectors.
If you've run something similar on a different corpus and got a different ranking, I'd like to hear it.
Citation
Niu et al. RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models. ACL 2024. arXiv:2401.00396.
Drafted by an AI assistant from the benchmark results and reviewed by me. Every number comes from results/metrics.json in the repo.
Top comments (10)
The interesting ablation is whether NLI earns its seven seconds inside the ensemble. If lexical plus numeric, both microseconds, reproduce the ensemble within noise, the neural detector is decoration - and that makes the headline stronger rather than weaker.
The second thing to report is the operating point instead of the curve. A hallucination check ships as a gate, so what matters is precision at the threshold you would actually run, and the costs are asymmetric: blocking a correct answer is visible, missing one is not. An AUC advantage that lives at 0.5 often disappears at 0.8.
And a lexical threshold is a per-corpus constant, so a small threshold sweep tells the reader whether your detector works or your tuning does.
Agreed on both. The ablation is the next thing I want in the repo: lexical + numeric alone vs. the full ensemble, per task, with bootstrap intervals, so it's visible whether NLI adds anything beyond noise. On operating points, you're right that a gate is judged at one threshold, not by AUROC. I'll add precision and false-block rate at a fixed recall for each detector, since blocking a correct answer and passing a hallucination don't cost the same.
Update, now in the repo (README, 'Ablation and operating points'): lexical+numeric scores AUROC 0.762 [0.734, 0.793] and the full ensemble 0.766 [0.739, 0.797]. The paired difference is +0.005 [-0.001, +0.011], so NLI adds nothing beyond noise, and the ensemble still doesn't beat lexical alone (0.778). At recall 0.8 lexical has precision 0.53 and false-block rate 0.40 (NLI 0.49 / 0.46). Your threshold point holds too: the recall-0.8 threshold is 0.23 on QA, 0.18 on Summary and 0.48 on Data2txt, and a threshold tuned on one task fails on the others. Thresholds are in-sample, as the README says.
The Data2txt drop where NLI falls to 0.496 while lexical overlap stays at 0.661 points to an overlooked flaw in how structured inputs get fed to entailment models.
Cross-encoder NLI models like DeBERTa are trained on grammatical premise-hypothesis sentences. When a source is flattened JSON or a serialized table without clear row-to-header prose bindings, the encoder has no reliable syntax to anchor entailment, so probabilities collapse toward noise. Lexical overlap ignores grammar completely and still catches fabricated entity names or attributes that never appeared in the source record.
On technical PDF and table RAG pipelines, keeping structured chunks as explicit Markdown tables or key-value pairs helps that cheap lexical check even further: it prevents normal formatting tokens from diluting content-token overlap, and makes the triage flag much easier for a human reviewer to verify against the original cell.
Ran the test: verbalising the Data2txt records before NLI did not close the gap. AUROC is 0.467 [0.398, 0.538] verbalised vs 0.496 raw JSON vs 0.661 lexical; verbalised minus raw is -0.029 [-0.084, +0.025], no detectable change. So it isn't only the input format. More likely the claims in those answers are numeric and attribute-level, where entailment models are weak anyway. Details and per-sample scores are in the repo.
That matches what I see in the per-sample errors: on Data2txt the source is serialized JSON, and the NLI model gets a "premise" with no sentence structure to anchor on. One cheap test is to verbalize the records first ("The restaurant X has rating 4.5") and rerun NLI. If the gap closes, it's an input-format problem rather than a model one. I'll try it and post the numbers in the repo.
Running the paired test was the right call, and the interval settles it: -0.016 [-0.027, -0.005] excludes zero, so numeric hurts on balance and the QA gain does not buy back the Summary loss.
On the C/(1+C) result, one reframe: that threshold is a property of blocking, not of the detectors. A block costs the user a failed answer; queueing the same case for async human review costs only reviewer minutes, which drops C by roughly an order of magnitude. At C = 0.5 the break-even precision is 0.33, and your 0.67 clears it easily.
So the finding is "these detectors cannot justify blocking at 5:1", which is a much sharper claim than "flag nothing" - and it points at a product decision rather than a modelling one.
Right, and the asymmetry has a number you can print: the cost ratio between the two errors. A false block usually costs more than a passed hallucination - the user notices, retries, or escalates - so the threshold should move with that ratio rather than sit at one recall target.
The output that makes it usable is one table: rows at cost ratio 1:1, 5:1, 20:1; columns precision and false-block rate at your fixed recall. Then a reader picks their row instead of arguing about AUROC.
Bootstrap intervals matter more on that table than on the aggregate, because the false-block rate is the sparse cell.
The thing I would change is how those three numbers are compared: 0.762, 0.766 and 0.778 all sit inside a CI roughly 0.06 wide, so the ranking between them is not resolvable. You cannot say lexical alone is best either. Only "NLI adds nothing" survives, because you tested that one as a paired difference with a narrow interval - and that is the right method.
Apply the same paired test to lexical versus lexical+numeric before dropping numeric. Your own method already hands you the comparison; the marginal CIs just cannot answer it.
And lead the table with false-block 0.40 at recall 0.8. That number decides whether this ships; AUROC is the one that hides it.
Ran both, now in the repo. Paired, lexical+numeric minus lexical is -0.016 [-0.027, -0.005] overall, so numeric hurts on balance (+0.024 on QA, -0.038 on Summary, about 0 on Data2txt). You were right that the marginal ranking wasn't resolvable; the paired one is.
Cost table, with the threshold re-selected to minimise C*FP + FN inside each bootstrap resample: at 1:1, lexical runs at 0.49 with precision 0.67 [0.61, 0.75] and false-block rate 0.14 [0.08, 0.24]. At 5:1 and 20:1 the cost-optimal policy for every detector is to flag nothing. Flagging only pays when precision exceeds C/(1+C), so 0.83 and 0.95, and none of them gets there on this data.
So as a hard gate this only makes sense where a false block costs about as much as a miss. Otherwise it's a triage flag for human review. The README now leads with the 0.40 false-block rate at recall 0.8; the full table is in results/cost_ratio_table.csv.