DEV Community: Hua Li

A true claim can still be a hallucination

Hua Li — Wed, 28 Oct 2026 05:36:01 +0000

Summary. Most discussion of hallucination is about claims that are false. A harder case is a claim that is true but not supported by the sources an answer cites. Summarization research calls this a factual hallucination, and grounding checkers used in production RAG systems flag it the same way they flag a false claim, because from the sources alone the two look identical. I ran into one in my own evaluation, on a question about a paper I co-wrote.

One of the 18 test questions in my RAG evaluation asks what cognitive biases one of my own papers addresses. The paper, from 2005, is about countering bias in search, but it never names a specific bias. Across three generations, the answer that said "the paper doesn't name specific biases" scored 0.00 on answer relevancy. The answer that confidently said "confirmation bias" scored 0.58. Faithfulness caught the unsupported claim and scored that answer 0.17. Then I found that the reference answer in my own golden set, which I had reviewed and marked verified, also said "confirmation bias." As the paper's co-author, I can say the claim is true: confirmation bias was one of the main problems the mechanism was built to address. It just isn't in the paper. This post is about the gap between a claim that is true and a claim the source supports: how current RAG systems and research handle it, a fifth approach I propose, and the case that prompted it.

Key points

A claim can be true and still unsupported by the sources an answer cites. A source-grounded check is right to flag it, but it can't tell it apart from a false claim.
Research already crosses source support with truth for every claim (RAGChecker, FRANQ). I'd add two refinements: keep claims that can't be checked either way in view, and split true-but-unsupported claims into derived and recalled (Figure 1).
Grounded derivation: let an answer go beyond its sources only by showing its steps from cited premises, with every outside premise named and drawn from a source declared in advance, and check every step (Section 3).
In my own evaluation, my answer key made the same unsupported claim as the model. The source-grounded judge caught what my review missed (Section 4).
RAGAS answer relevancy scored the honest answer, "the paper doesn't name specific biases," at 0.00, and the confident unsupported one at 0.58.

Figure 1. Every claim in a RAG answer has two properties: whether the source supports it, and whether it can be verified as true, false, or neither. The supported/unsupported × true/false grid follows RAGChecker (2024) and FRANQ (2026); keeping the unresolved column in view and splitting derived from recalled are the refinements proposed in Section 3. The dots mark the claim at the center of this post: as "the authors intended to target confirmation bias," it is unresolved from the documents; as "the method is best read as addressing confirmation bias," it is derived, given one named outside premise.

1. Background: when a hallucination is true

A RAG system has two goals that usually agree:

Correctness: is the answer true?
Verifiability: can a reader check it against the sources it cites?

Retrieval exists mainly for the second: every claim can be traced to a document. Evaluation follows the same split. Faithfulness (also called groundedness) asks whether each claim is supported by the retrieved text. Correctness or factuality asks whether the claim is true, usually by comparing it with a reference answer.

The two can come apart. Maynez et al. studied this in abstractive summarization in 2020 and separated faithfulness to the source document from factuality in the world. They defined a factual hallucination as content that is not in the source but is still correct. It was the minority case: over 90% of the content their annotators found added beyond the source was wrong. That is exactly what makes the true cases hard. A checker that sees only the sources finds a claim it can't support, and it has no way to tell the rare true claim from the common false one. Most production systems resolve the conflict in favor of verifiability: if the sources don't support a claim, it counts as a failure, true or not. My evaluation did the same, and it was right to. What I didn't expect was that I had written the same unsupported claim into my own answer key.

2. How the field handles this

There is no general solution. Current systems and research take four approaches, each with a cost. Section 3 proposes a fifth.

Strict grounding: unsupported counts as wrong. This is the default for enterprise and regulated uses. The generator is told to answer only from the provided context, and a grounding checker scores each claim against the sources. Google's check-grounding API says that "every claim in the answer candidate must be supported by one or more of the given facts," and Microsoft's groundedness detection is likewise defined relative to the source material. My faithfulness metric works the same way. The cost: a true claim and a false one get the same verdict, as the confirmation-bias answer did in my evaluation.

Measure faithfulness and correctness separately. Evaluation frameworks, RAGAS included, have both a source-based metric and a reference-based one. Together they can separate the four cases: true and supported, true and unsupported, false and supported (the source itself is wrong), false and unsupported. RAGChecker (NeurIPS 2024) reports exactly this split for every claim in an answer: claims that are supported and correct, supported but incorrect ("noise sensitivity"), unsupported but correct ("self-knowledge"), and neither ("hallucination"). FRANQ (Findings of ACL 2026) builds its scoring on the same grid (the first two columns of Figure 1), and starts from the same complaint as this post: correct statements labeled as hallucinations because the retrieved text doesn't support them. The weak point is the reference. A correctness metric is only as good as the answer key, and mine contained the same unsupported claim as the answer it would have graded, so it would have scored the confirmation-bias answer as correct.

Model the conflict between memory and source. Research on knowledge conflicts shows the problem cuts both ways. ClashEval (2024) found that models adopt incorrect retrieved content over 60% of the time, overriding a correct prior: trusting the source too much. My case is the opposite, trusting memory too much. Methods such as FaithfulRAG (ACL 2025) model conflicts at the level of individual facts, so a system can notice when what it knows and what it retrieved disagree, instead of silently picking one.

Allow general knowledge, but label it. Every sentence either carries a citation or is marked as coming from the model's general knowledge, so the reader can see which parts are checkable. Self-RAG (ICLR 2024) builds a version of this into the model, which predicts whether each part of its output is supported by the retrieved passage. For the question in my evaluation, a labeled answer would have read: "The paper doesn't name specific biases [cited]. Based on its method, it is best read as addressing biases such as confirmation bias (not stated in the source)." That answer is honest, useful and checkable, and it is not what any of the answers my system generated said.

Which approach is right depends on what an unsupported claim costs. In a research assistant, a labeled inference is helpful. Anywhere decisions have to be audited, such as claims review, compliance or legal work, a flag the system can't trace to a document can't be relied on, even when it happens to be correct.

3. A fifth approach: grounded derivation

The four approaches above treat support almost as a lookup. The RAGAS faithfulness judge accepts a statement if it "can be directly inferred based on the context": one step of inference, never shown or checked.

I want to propose a different standard. Language models are good at reasoning. So instead of asking only whether a claim is stated in the sources, ask whether it can be derived from them, and require the answer to show how. Every claim in an answer then falls into one of three kinds:

Stated: supported directly by a cited passage.
Derived: reached by explicit reasoning steps whose premises are cited passages.
General knowledge: a premise or claim that is not in the sources, labeled as such.

An evaluator then checks three things: that each citation supports its premise, that each step follows from the ones before it, and which premises came from outside the sources. "Is this conclusion supported?" is hard to judge when the conclusion isn't in the text. "Does step 3 follow from steps 1 and 2?" is a small, local question, like checking a proof line by line.

The running example is the claim from the summary: that my 2005 paper addresses confirmation bias. The paper describes an anti-bias mechanism, ABM, for a search system used by intelligence analysts, based on Richards Heuer's Analysis of Competing Hypotheses (details in Section 4.1). It is worth being precise about the claim first, because it can mean four different things:

The method could reduce confirmation bias.
The method is best read as addressing confirmation bias.
The authors intended it to counter confirmation bias.
The paper names confirmation bias.

The first two can be argued from the paper plus general knowledge about Heuer's method, by anyone who reads it; the derivation below shows how. The third is true, but its only support is my testimony as an author. The fourth is false. The answer my system generated asserted the fourth, which is why faithfulness was right to reject it. My answer key asserted something between the third and the fourth.

Here is version 2 as a grounded derivation:

ABM is inspired by Heuer's Analysis of Competing Hypotheses. (stated in the paper)
ABM retrieves results for competing versions of the analyst's query and mixes them into the analyst's results. (stated in the paper)
Analysis of Competing Hypotheses is designed to counter the tendency to look only for evidence that confirms a hypothesis, that is, confirmation bias. (general knowledge: from Heuer's book, not from this paper)
Therefore ABM is best read as addressing confirmation bias. (derived from 1–3, as an inference to the best explanation)

The derivation does not make the claim supported. It needs premise 3, which the paper doesn't contain. What it does is replace one vague verdict ("unsupported") with a precise one: the conclusion follows, given a single named outside premise that a reader can check in Heuer's book. Had my corpus included Heuer, the whole claim would have been derived from sources. This is also what my verification of the answer key should have looked like. Note what the derivation does not reach: it supports version 2 of the claim, not version 3. No derivation from the paper can establish what its authors intended.

Three refinements matter.

Deduction is not the same as inference. In a deduction, true premises make the conclusion certain: "therefore." In an inference to the best explanation, the conclusion is only the most likely reading: "most likely." Claims about purpose or intent, like this one, are almost always the second kind. A derived claim should say which kind it is, so a reasonable inference doesn't carry the authority of a proof.

The reasoning has to be checked, not trusted. Language models can write chains that look rigorous and hide a leap or an unstated assumption, and an LLM judge tends to accept a chain whose conclusion it already believes. Each step needs a strict check, and every added step adds risk. The longer the chain, the more cautious the label should be.

Outside premises can't be invented to fit. Any unsupported claim H can be "derived" from a stated premise P by adding the outside premise "P implies H." That premise is doing all the work, and accepting it would make every hallucination derivable. So an outside premise counts only if it is a general rule or definition that doesn't mention the specific document or claim, and if it comes from a source declared in advance: a named textbook or reference work, a statute, a standard definition, or a program that can check it. Step 3 above passes: it is a general statement about Heuer's method, checkable in Heuer's book.

Where it works best: mathematics and law. A source doesn't need to say 2 + 2 = 4; an answer can derive it from the axioms of arithmetic. Axioms are outside premises of an established kind: definitions, proven theorems and statute text can be used freely, but should still be named. Contestable premises, like step 3 above, must be labeled. In mathematics a program, such as a computer algebra system or the proof assistant Lean, can check each step, so the check itself can't hallucinate. Law teaches the other lesson. Applying a statute to facts reads like a proof, but interpretation, precedent and jurisdiction make most legal conclusions "most likely," not "therefore." Insurance claims review is similar: a coverage decision applies policy wording to the facts in a claim file.

Parts of this idea exist already. EntailmentBank (2021) explains answers as entailment trees built up from premises. ProofWriter (2021) finds the missing fact that would make a conclusion provable, in a controlled logical setting. PARC (ICML 2025) links each step of a math solution to the earlier steps it depends on. AttrScore (2023) labels a claim "extrapolatory" when the source lacks enough information to support it, without separating derivable claims from unsupported ones. MiniCheck trains grounding checkers to recognize a fact assembled from several sentences. RLSeek (ACL 2026) makes a hallucination detector quote the source at each step of its reasoning. RAGChecker counts correct claims that no retrieved chunk supports, as "self-knowledge." Work on abductive RAG generates and validates premises that the evidence is missing. What I have not found in published RAG evaluation is the combination: answers that label every claim as stated, derived or general knowledge, with each derivation shown and each outside premise named, and an evaluation that checks every part. I plan to test it on my own corpus (Section 9).

Figure 1 puts the field's grid and this proposal together, with two refinements:

Keep unresolved claims in view. FRANQ's annotation already labels some claims "unverifiable," then drops them from evaluation. Reporting them, along with how much of an answer could be checked at all, stops "can't be checked" from being counted as either true or false.
"Derived" is a kind of true-but-unsupported claim, not the whole category. That box also holds plain recall, as when an answer adds a fact from the model's memory. Grounded derivation is what tells the two apart.

This post's title follows Maynez's usage: a claim the source doesn't support is a hallucination, even when it is true. What the grid adds is a way to say which kind.

4. The case: a question about my own paper

4.1 The question and the paper

My test system is a small RAG service over 15 of my research papers, evaluated with RAGAS 0.2.15. Generator: gpt-oss-20b on Groq. Judge: Gemini 3.5 Flash Lite. Retrieval: top 8 chunks. The setup is described in part 1 and part 2 of this series.

Question cont-05 reads:

What cognitive biases affect information retrieval according to this research?

The source is a two-page paper Rafael Alonso and I published at the 2005 International Conference on Intelligence Analysis, Combating Cognitive Biases in Information Retrieval. It describes an anti-bias mechanism for a search system used by intelligence analysts. The idea comes from Richards Heuer's Analysis of Competing Hypotheses. The system automatically generates alternative versions of the analyst's query, for example by swapping concepts for WordNet antonyms, coordinate terms or hypernyms. It runs those alternative queries too, and mixes a few of their best results into what the analyst sees. The paper calls them "anti-bias" results.

What the paper does not do is name a bias. It says that queries "inherit cognitive biases" from the models that produce them. It never says "confirmation bias," "anchoring" or any other specific bias. I checked the full text for this post.

So the honest answer to the question is roughly: the paper treats cognitive bias in general and doesn't name specific biases; its contribution is a mechanism for countering them.

4.2 Three answers, three verdicts

Part 2 generated three sets of answers under identical settings. Retrieval returned the same 8 chunks every time; only the generated text changed. Here is cont-05:

Generation	What the answer said	Faithfulness	Answer relevancy
1	The paper addresses cognitive bias in general; the excerpts don't list specific biases	0.80 (8 of 10 claims supported)	0.00
2	"Confirmation bias is explicitly addressed"; defines it; says the method "explicitly guards against confirmation bias"	0.17 (2 of 12 claims supported)	0.58
3	The paper speaks of biases "in general terms"; the excerpts "do not list specific bias names"	0.73 (11 of 15)	0.87

Faithfulness worked. It splits the answer into claims and checks each one against the retrieved chunks. On generation 2 the judge rejected every confirmation-bias claim, with reasons like "The term confirmation bias does not appear in the available text excerpts." The two claims it accepted were the parts copied from the paper: the method creates alternative maps and selects anti-bias results.

Answer relevancy punished the honest answer. RAGAS relevancy asks the judge to write questions that the answer seems to respond to, then compares them with the real question by embedding similarity. It also asks the judge whether the answer is "noncommittal," and in RAGAS 0.2.15 a single noncommittal flag among the generated questions sets the score to zero (score = cosine_sim.mean() * int(not committal)). Generation 1 said the excerpts don't name specific biases. The judge flagged that as noncommittal in two of three samples, so the answer scored 0. It scored 0 again in both re-judging passes.

Generation 3 said almost the same thing and scored 0.87. My best guess is framing: generation 3 opened with a confident sentence about which papers discuss bias and put the "no specific names" point in the middle, while generation 1 led with the hedge.

Read alone, relevancy would rank the confident wrong answer above the honest one. Faithfulness ranks them correctly. Neither metric is enough by itself.

4.3 My answer key had the same bias

Here is the reference answer for cont-05 in my golden set, marked "verified": true:

The 2005 work examines how analyst cognitive biases such as confirmation bias distort information seeking and retrieval, and proposes system-side mechanisms to counteract them during search.

The generator and I made the same claim, and only one of us was being checked against the source. When I verified the reference, I checked that it described the paper correctly. It did. I didn't check that each phrase appeared in the paper, and one didn't.

The reference feeds context recall, which asks the judge whether each reference statement is supported by the retrieved chunks. This reference was a single sentence, so recall on cont-05 was one yes/no call. In part 2's three re-judging passes, on identical inputs, the judge said no, no, then yes. The "yes" is worth quoting:

"...though confirmation bias specifically is not named in the text, the overall premise of addressing cognitive biases in information retrieval is present."

That is the judge accepting a plausible inference because the rest of the sentence fit, the same move I made when I reviewed the reference.

I found the problem while splitting single-sentence references into several checkable claims, a fix from part 2. To split a reference into claims, you have to find each claim's sentence in the source. "Such as confirmation bias" had no sentence. The rewritten reference, now in the repo's golden set, has three claims, each checked against the paper's text:

The 2005 work presents an anti-bias mechanism (ABM) for combating cognitive biases in information retrieval. ABM automatically generates alternative, competing queries alongside the user's original query. A small set of the best results from the competing queries is mixed in with the original results.

4.4 True, but not in the source

The claim isn't a mistake about the work. When Rafael and I designed the anti-bias mechanism, confirmation bias was one of the main biases we had in mind. Intelligence analysts are prone to it: once a hypothesis forms, they look for and weigh evidence that supports it, and it is notoriously hard to counter. Section 3 shows how the mechanism connects to it. The paper doesn't say any of this because it was a two-page conference paper. We described the mechanism and left out the discussion of which biases it targeted.

In the terms of Section 3, generation 2 claimed version 4, and my answer key something between versions 3 and 4. So the generator in generation 2 was, in a sense, right about the work. It was still wrong about the source, and in a RAG system the source is what counts. The system's promise is "here is what these documents say," and the documents don't say this. A reader who followed the citation would not find it. The generator had no access to my memory of 2005; it filled the gap with the most plausible bias.

It slipped past me because it was true, familiar, and consistent with the rest of the sentence. When I verified the reference, I checked it against what I knew about the paper, which includes things the paper doesn't say. The faithfulness judge checked each claim against the text, and the text was all it had. For a source-grounded system, that is the right standard, even when the author knows more.

There is an irony here. The 2005 paper argues that a search system inherits the biases of the person who writes the query. My evaluation inherited my knowledge, and my blind spot, through the answer key.

5. Beyond faithfulness: a preview of part 4

Grounded derivation also exposes a gap in the RAGAS metrics I use: they ask whether an answer matches its sources and its question, not whether it is true. Suppose a source has a typo: 2 + 2 = 5. An answer that copies it scores perfectly on faithfulness and is wrong. An answer that says 4 is scored as unsupported and is right.

When a source is wrong, the best answer is neither faithful nor silently corrected. It is both: "The source states 2 + 2 = 5; this appears to be an error: 2 + 2 = 4." Scoring that answer fairly is harder than it looks. "The source states 2 + 2 = 5" is true, while "2 + 2 = 5" is false, so an evaluator that strips "the source states" penalizes exactly the answer it should reward. That is where part 4 will start.

6. A correction to part 2

Part 2 said "confirmation bias" "isn't in the corpus." It is: a 2013 paper in the same corpus, about a serious game that teaches players to recognize their own biases, names it. The accurate statement is narrower: it isn't in the 2005 paper, and every judge explanation says it wasn't in the excerpts retrieved for this question. That exposes an ambiguity, too. "According to this research" could mean the 2005 paper or my work as a whole, and under the second reading "confirmation bias" is a supported answer that retrieval missed. My harness doesn't log retrieved document IDs yet, so the judge explanations are my only evidence about which papers were retrieved.

7. What I do now

Check references against the source, one claim at a time. For each claim in a reference answer, find the sentence that supports it. If you can't point to one, the claim came from you, not the document, even if it is true. Reading a reference and nodding along is not verification. Domain experts and authors are the most exposed, because they know more than the documents say.
If the knowledge matters, put it in the corpus. If users should learn what a paper was designed to address, that has to be written down somewhere the system can retrieve, not supplied by the generator.
Ask for labeled inferences instead of forbidding them. If the generator may use general knowledge, have it mark which sentences are not from the sources. A labeled inference can be judged on its own terms; an unlabeled one can only be scored as unsupported.
Read faithfulness and relevancy together. High relevancy with low faithfulness is a confident answer the sources don't support. Low relevancy with high faithfulness may be an honest abstention, which RAGAS relevancy scores as zero.
Scope questions explicitly. "This research" should say which paper, or say "across the corpus." An ambiguous question makes the right answer depend on the reader.
Log retrieved document IDs. Without them, "was the supporting paper retrieved?" has to be answered by reading judge explanations.
Write a competing reference for claims you're sure of. This is the 2005 paper's idea applied to evaluation: for each claim you're confident about, ask what the source would need to say for it to be false, and check whether it says that. That was how "such as confirmation bias" should have been caught.

8. Limits

This is one question out of 18, so treat it as a case study, not a measurement. The scores come from single runs except where re-judging is mentioned. The judge's explanations are themselves LLM output and can be wrong. They are consistent here, and they match the paper's text. That confirmation bias was a target of the 2005 mechanism is my account as an author, not something the paper states, which is the point of this post.

9. Next

The other 12 rewritten multi-claim references go through the same check against the papers before I merge them. Then I'll re-judge the saved results against them to see whether recall becomes less noisy. I'll also add document-ID logging, so the next post can say which papers were retrieved instead of inferring it.

Then I want to test grounded derivation. As the author of every paper in the corpus, I can propose claims that are true but unstated, but my knowledge can't be the answer key, for the reason this post is about. So each claim would be labeled twice: by me, and by a reader who didn't write the papers and has only the papers and standard references. Where the reader agrees with me, the claim is a genuine derivation; where only I know it, it is author intent, which an evaluator shouldn't reward. Then the question is whether a derivation-checking evaluator can tell those cases apart, including on deliberately flawed derivations designed to fool it.

Part 4 will look at how evaluators score an answer that flags and corrects a source error.

The harness, the golden set and every raw judge output are in the repo: github.com/huali10044/research-rag.

References

Alonso, R., & Li, H. (2005). Combating Cognitive Biases in Information Retrieval. Proceedings of the International Conference on Intelligence Analysis.
Heuer, R. J. (1999). Psychology of Intelligence Analysis. Center for the Study of Intelligence, Central Intelligence Agency.
Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020). On Faithfulness and Factuality in Abstractive Summarization. ACL 2020.
Yue, X., et al. (2023). Automatic Evaluation of Attribution by Large Language Models (AttrScore). Findings of EMNLP 2023.
Dalvi, B., et al. (2021). Explaining Answers with Entailment Trees. EMNLP 2021.
Tang, L., Laban, P., & Durrett, G. (2024). MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents. EMNLP 2024.
Ru, D., et al. (2024). RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation. NeurIPS 2024 Datasets and Benchmarks.
Huang, Z., et al. (2026). RLSeek: Evidence-Grounded Reasoning for RAG Hallucination Detection. ACL 2026.
Lin, S. (2025). Abductive Inference in Retrieval-Augmented Language Models: Generating and Validating Missing Premises. arXiv preprint.
Fadeeva, E., et al. (2026). Faithfulness-Aware Uncertainty Quantification for Fact-Checking the Output of Retrieval Augmented Generation (FRANQ). Findings of ACL 2026.
Mukherjee, S., et al. (2025). Premise-Augmented Reasoning Chains Improve Error Identification in Math Reasoning with LLMs (PARC). ICML 2025.
Tafjord, O., Dalvi, B., & Clark, P. (2021). ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Language. Findings of ACL 2021.
Wu, K., Wu, E., & Zou, J. (2024). ClashEval: Quantifying the tug-of-war between an LLM's internal prior and external evidence.
FaithfulRAG: Fact-Level Conflict Modeling for Context-Faithful Retrieval-Augmented Generation. ACL 2025.
Asai, A., Wu, Z., Wang, Y., Sil, A., & Hajishirzi, H. (2024). Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection. ICLR 2024.
Google Cloud. Check grounding with RAG.
Microsoft. Groundedness detection.
RAGAS 0.2.15, _answer_relevance.py (noncommittal scoring rule).

Judge noise or answer noise? What really moves RAG evaluation scores

Hua Li — Mon, 19 Oct 2026 04:02:06 +0000

Summary. RAG evaluation scores change from run to run even when nothing in the system changes. I measured where that variation comes from in my own system: from the LLM judge grading the same answers differently, or from the generator writing different answers. For faithfulness, about 85% of the run-to-run variance came from the answers. For answer relevancy, nearly all of it did. Context recall and context precision never read the generated answer, so with deterministic retrieval all of their variance came from the judge, even with the judge set to its most consistent setting. From these measurements, a difference between two settings, each run once on my 18 questions, needs to exceed about 0.15 (faithfulness), 0.08 (relevancy), 0.09 (recall) or 0.10 (precision) to mean anything. Applying that bar to the top-k sweep in my first post confirms that the sweep could not rank the k values, and that the "optimal k=8" came from uneven question coverage and single runs.

Figure 1. Gray dashed line: the recall curve published in the first post (one run per k, unequal question sets). Blue dots: every run of that sweep, scored on the 9 questions all seven runs scored. Shaded band: where a single run's 9-question recall lands from judge noise alone (mean ± 0.09).

Key findings

The judge does not always give the same verdict twice, even when set to be as consistent as possible. Grading the same answers three times moved the 18-question means by 0.003 (relevancy) to 0.085 (precision).
For faithfulness and relevancy, the answers matter more than the judge. Run-mean standard deviation rose from 0.021 to 0.054 for faithfulness and from 0.002 to 0.028 for relevancy when the answers were regenerated.
Recall and precision are pure judge noise in this setup. RAGAS computes them from the question, the retrieved chunks and the reference answer. Regenerating answers cannot change them.
Noise bars for comparing two single runs: about 0.15 faithfulness, 0.08 relevancy, 0.09 recall, 0.10 precision (18 questions; about 40% wider on 9).
The first post's k=8 peak does not survive. On the 9 questions every run scored, all seven sweep runs fall inside the judge-noise band (Figure 1). The first post's claim that changing the judge model moves scores more than changing k gets stronger.
Replicate the right thing. For recall and precision, re-judge saved results. For faithfulness and relevancy, regenerate answers. Re-judging alone would make faithfulness look more than twice as stable as it is. ## 1. Background

In the first post I swept top-k over a small RAG system, found a clean winner at k=8, then ran k=8 twice more and watched the winner dissolve: same configuration, recall anywhere from 0.61 to 0.83.

A commenter, Ahmet Özel, asked the obvious next question: how much of that spread is the judge, and how much is the generator writing different answers? The two need different fixes. If the judge is noisy, you grade more times. If the answers are noisy, you generate more times. I had been doing neither, and I couldn't tell which one I needed.

2. Setup

Same system as before: 15 research papers, 18 judged questions, k=8. The generator is gpt-oss-20b on Groq. The judge is Gemini 3.5 Flash Lite, through RAGAS 0.2.15 with four metrics: faithfulness, answer relevancy, context precision and context recall.

One setting matters for what follows. An LLM's temperature controls how much randomness goes into choosing each word: higher values give more varied output, and 0 tells the model to always pick its most likely next word, which in principle makes the same input produce the same output. My harness sets the judge's temperature to 0, the usual advice for LLM judges. The generator runs at the provider's default temperature, so its answers are expected to vary from run to run.

Two experiments:

Judge only. Generate one set of answers, save them with their retrieved chunks, and have the judge score the same saved answers three times. Nothing is regenerated or re-retrieved. The prompts sent to the judge had identical hashes in all three passes, so any difference is the judge.
Generator plus judge. Generate three separate sets of answers under identical settings and score each once. To make the first experiment possible, the harness had to learn to replay: save every answer and its retrieved chunks, then re-run only the judge. It now also records why any score is missing (rate limit, timeout, unparseable judge output and so on) and reports each mean with its coverage, like 0.833 (9/18). That second change is how I found out that the 0.833 in my first post covered only half the questions.

3. Results

3.1 The judge alone

Same answers, same chunks, same prompts, three passes:

Metric	Pass 1	Pass 2	Pass 3	Range
Faithfulness	0.653	0.611	0.628	0.042
Answer relevancy	0.821	0.824	0.824	0.003
Context precision	0.601	0.686	0.632	0.085
Context recall	0.694	0.648	0.667	0.046

Setting the judge's temperature to 0 did not make it repeat itself. Each metric varies for a reason specific to how it's built.

Faithfulness splits the answer into claims, then checks each claim against the chunks. The judge split identical answers into different numbers of claims: one answer came out as 5, 10 and 11 claims in the three passes. The score is supported claims over total claims, so the denominator moves. Faithfulness changed on 11 of 18 questions, and those were exactly the 11 where the claim count changed.

Precision and recall are built from yes/no verdicts. 5 of 148 chunk verdicts flipped between passes, and 3 of 22 reference-statement verdicts. That sounds small until you notice that 14 of my 18 reference answers were a single sentence. For those questions, recall is one yes/no call, so a single flip moves that question from 1 to 0. One question went 1, 0, 0 across three identical passes.

Relevancy barely moved. The judge writes questions the answer seems to address, and an embedding model compares them with the real question. The embedding step smooths out the judge's wording.

3.2 Adding new answers

Retrieval returned exactly the same 8 chunks for every question in all three generations, so the only thing that changed was the generated text, and it changed a lot: most answers to content questions shared less than a fifth of their wording with the previous generation.

Spread of the 18-question mean, as a standard deviation across runs:

Metric	Judge only	New answers, each judged once	Share from the answers
Faithfulness	0.021	0.054	about 85%
Answer relevancy	0.002	0.028	nearly all
Context recall	0.032	—	none
Context precision	0.035	—	none

The blanks are the most useful finding of the whole exercise. In RAGAS, context recall and context precision never read the generated answer. They compare the retrieved chunks with the question and the reference answer. My retrieval is deterministic, so regenerating answers cannot change those two scores, and all of their run-to-run noise is the judge.

I would have guessed the opposite split before running this. I'd assumed the judge was the main problem everywhere.

3.3 How big a difference needs to be

Compare two settings, say k=5 against k=8, each run once on these 18 questions. Each run's mean has its own noise, the difference between two runs has about 1.4 times that, and the difference should be about twice its own noise before you believe it:

Metric	Smallest difference worth reporting
Faithfulness	about 0.15
Answer relevancy	about 0.08
Context recall	about 0.09
Context precision	about 0.10

Each bar works like a 95% interval on the difference between two runs. They come from three or four runs per metric, so treat them as rough; the true values could plausibly be half or double. More runs would tighten them, but each full run costs about 257 judge calls against a free-tier limit of 500 a day. On a 9-question subset the bars are about 40% wider.

These bars treat the two runs as independent. When both settings are scored on the same questions by the same judge, the noise is probably partly shared and cancels in the difference, so a paired comparison would likely give tighter bars.

4. What this changes in my first post

Figure 1 puts the first post's sweep against the measured noise band. Going through that post's conclusions one at a time:

"k=8 is the optimum." Gone, and now I know why. Two things built that peak. First, the runs were scored on different questions: the k=8 run behind the 0.83 had only 9 of 18 questions scored, because the judge failed on the rest, and the questions it failed on were the harder ones. Second, each setting was run once. Put every run on the same 9 questions and the curve flattens to between 0.67 and 0.83. All seven runs fit inside the band, so nothing in the sweep can be told apart from noise. Only three of those nine questions changed score in any run (cont-01, cont-03 and meta-05); the other six scored the same in all seven.

"Noise exceeds signal on every metric." Still true, but I overstated the noise. The first post said recall varied by 0.22 across three k=8 runs. Part of that was the same coverage problem: on matched questions the k=8 runs spread by 0.11. The conclusion stands because the between-setting differences shrank too. What's new is knowing where the noise comes from: for recall and precision it's all the judge, and for faithfulness it's mostly the answers.

"k=3 under-retrieves." Weakened to a hint. k=3 still has the lowest recall on the matched questions (0.67, against about 0.77 for the other runs on average), and it sits on the lower edge of the band. That's consistent with retrieving too little, but the gap is below the 0.13 bar for 9 questions, and all of it comes from one question (cont-01), which k=3 missed and every other run found. I'd now call it a direction worth testing, not a finding.

"The judge moves the numbers more than the parameter does." Stronger. I hedged this last time because I had only one run per judge and couldn't separate a judge effect from an unlucky draw. Now I know the same judge's run-to-run noise: the bar is about 0.09 for recall and 0.15 for faithfulness. Switching judge models moved recall by 0.31 and faithfulness by 0.26: more than three times the bar for recall, and nearly twice the bar for faithfulness. One caveat from the first post still applies: the two judges failed on different questions, so part of that gap is coverage. But it is too large to be run-to-run noise.

"The significance test couldn't find anything." Now I know why so many pairs were tied. The first post blamed Wilcoxon's floor for dropping tied pairs: 8 of 9 pairs scored identically between k=8 and k=12. The underlying reason is the references. 14 of my 18 reference answers were a single sentence, so recall on those questions can only be 0 or 1. Any two settings that retrieve the same relevant chunk tie exactly, and the test has nothing left to work with.

"Replicate before believing." Still right, but replicate the right thing. For recall and precision, re-judge the same saved results; it's also cheaper, since there are no generation calls. For faithfulness and relevancy, regenerate the answers, because re-judging captures only about 15% of faithfulness's variance.

5. Failure cases the per-question view showed

Faithfulness handles a correct "no" badly. One question asks which paper won a best-paper award. None did, and all three generations said so correctly. Faithfulness scored them 1.00, 0.33 and 0.00. The "no" itself isn't checkable against the chunks, so the score depends on whatever incidental sentences surround it.

Relevancy rewards confidence, including invented confidence. One question asks which cognitive biases a paper addresses. The papers don't name specific biases. One generation said so and scored 0 on relevancy, because the metric treats a hedged answer as noncommittal. Another generation confidently listed "confirmation bias", which isn’t in the 2005 paper or in the excerpts retrieved for this question, and scored 0.58. Faithfulness caught the invention (0.17), which is the argument for never reading relevancy alone.

Missing scores have causes, and the causes are fixable. Across about 1,200 judge calls, two failed because the judge wrote malformed JSON escapes. On one day the judge slowed to about ten times its usual response time, and context precision timed out on 16 of 18 questions while the other three metrics finished. The cause: RAGAS's time limit covers a whole metric on one question, not each call, and precision makes one judge call per retrieved chunk (8 at k=8, against one to three for the others). A slow judge hits precision first, every time. Without per-reason logging, all of that would have shown up as NaN and silently shrunk the average.

6. Recommendations

Freeze the question list before scoring, and report every row. Fix the list of question IDs before any run, compare runs only on that list, and show all of them, with the reason for each missing score, so a complete-set result is never confused with a smaller matched subset. Appendix A shows this for the three generations. (Suggested by Ahmet Özel in a follow-up comment.)
Report coverage with every mean. 0.833 and 0.833 (9/18) are different claims.
Where you can, score retrieval without an LLM. If each question is labeled with the documents that should be retrieved, recall@k and similar measures can be computed by simple counting. With deterministic retrieval they would have no run-to-run noise at all, which is exactly where the judge-based recall and precision are weakest here. (Suggested by another reader.)
Know which metrics read the answer. Retrieval metrics need re-judging; answer metrics need regenerating.
Compute a noise bar before comparing settings. Two or three runs per setting are enough to estimate one.
Write references as several checkable claims. A one-sentence reference turns recall into a coin flip.
Name the judge, its version and the library version next to every number.

7. Limits

One corpus of 15 papers, 18 questions, one value of k, one generator and one judge. The variance estimates come from three or four runs each. The September sweep used a Gemini judge whose exact model version wasn't recorded, so the noise band in Figure 1 is borrowed from October's judge and the re-reading is indicative, not exact. The thresholds are specific to this setup; the method for getting them is not. The 85% answer share is likewise specific to this generator, prompt, judge and question set.

8. Next

I've rewritten the 13 single-sentence references as two to four claims each. Re-judging the saved results against them will show whether multi-claim references reduce the judge noise on recall, and by how much. Next, I'll label each question with the papers that should be retrieved and log the retrieved document IDs on every run; my harness currently saves the retrieved text but not the IDs. That gives a judge-free retrieval score to set beside the judge-based one. After that, the top-k sweep again, with enough replicates per setting to clear the bar.

The harness, the result files and every raw judge output are in the repo: github.com/huali10044/research-rag.

Appendix A. All 18 questions, three generations

Faithfulness and answer relevancy for each generation, each judged once. Every question is listed; a missing score shows its recorded reason.

Question	Faithfulness G1	G2	G3	Relevancy G1	G2	G3
`meta-01`	1.00	0.83	1.00	0.94	0.97	— (judge output unreadable)
`meta-02`	0.75	1.00	1.00	0.97	0.98	0.98
`meta-03`	1.00	1.00	0.89	0.78	0.78	0.85
`meta-04`	0.33	1.00	1.00	0.95	0.94	0.96
`meta-05`	0.50	0.50	0.33	0.72	0.67	0.70
`cont-01`	1.00	0.60	0.90	0.97	0.78	0.70
`cont-02`	0.80	0.78	0.79	0.99	0.99	0.99
`cont-03`	0.47	0.23	0.33	0.83	0.90	0.99
`cont-04`	0.26	0.00	0.12	0.96	0.98	0.80
`cont-05`	0.80	0.17	0.73	0.00	0.58	0.87
`cont-06`	0.43	0.38	0.38	0.93	0.89	0.88
`cont-07`	0.61	0.25	0.33	0.73	0.81	0.78
`cont-08`	0.56	0.52	0.43	0.98	1.00	1.00
`cont-09`	0.80	1.00	1.00	0.71	0.67	0.61
`cont-10`	0.29	0.22	0.40	0.64	0.85	0.76
`cont-11`	0.56	0.28	0.45	0.70	0.87	0.58
`adv-01`	0.60	0.82	0.14	0.84	0.98	0.83
`adv-02`	1.00	0.33	0.00	0.95	0.95	0.87

I found the optimal top-k for my RAG system. Then I ran it again.

Hua Li — Wed, 07 Oct 2026 13:00:00 +0000

I built a small retrieval-augmented generation system over a corpus of research papers and wanted to answer an ordinary question: how many chunks should I retrieve? So I did what the tutorials suggest — swept top-k across 3, 5, 8 and 12, scored each configuration with RAGAS, and read off the winner.

The answer was clean. k=8 led on all four metrics. Context recall was 0.53 at k=3, peaked at 0.83 at k=8, and fell back to 0.64 at k=12 — a textbook inverted U, with a story to match. Too few chunks and the generator starves; too many and distractors dilute the prompt. I had a number, a mechanism, and a plot.

Then I ran k=8 twice more, changing nothing.

Run	Faithfulness	Answer relevancy	Context precision	Context recall
k=8, run 1	0.767	0.900	0.655	0.833
k=8, run 2	0.590	0.804	0.585	0.611
k=8, run 3	0.640	0.835	0.687	0.625

Recall ranged from 0.611 to 0.833 across three runs of an identical configuration. The 0.833 that made k=8 look like a peak was the high draw of three.

Set the variation within one configuration against the differences between configurations:

Metric	Spread between configs (means)	Spread within k=8
faithfulness	0.084	0.177	noise exceeds signal
answer relevancy	0.059	0.096	noise exceeds signal
context precision	0.023	0.102	noise exceeds signal
context recall	0.162	0.222	noise exceeds signal

On every metric, one configuration varies more against itself than the configurations vary against each other. k=5, k=8 and k=12 are not distinguishable on this evidence. The only ordering that survives is that k=3 under-retrieves — it has the lowest recall in every run.

The inverted U was an artifact of running each configuration once.

The judge moves the numbers more than the parameter does

While building the harness I had changed judge models — the free tier I started on ran out of quota. That turned out to be a second experiment I hadn't meant to run.

Same corpus, same generator, same k=5. Only the judge differs:

Metric	Judge A	Judge B	Difference
faithfulness	0.840	0.582	0.258
context recall	0.917	0.604	0.313
context precision	0.752	0.628	0.125
answer relevancy	0.777	0.861	0.084

Changing the judge moved faithfulness by 0.26 and recall by 0.31. Changing top-k from 3 to 8 — the parameter I was actually studying — moved faithfulness by 0.12.

Part of that gap is different grading. Part is something subtler: the two judges fail differently. One of them intermittently returned output the parser couldn't read, so those samples scored as missing and dropped out of the average. The two numbers were therefore computed over different subsets of questions. Both effects are properties of the judge, and both are invisible in a reported score.

I want to be careful here: I have one run per judge, so I can't cleanly separate a judge effect from an unlucky draw. The honest statement is that the judge shift is about the same size as the run-to-run variation I measured, which is already enough to make absolute RAGAS numbers meaningless without naming the judge that produced them.

The significance test couldn't have found anything

Comparing configuration means throws away the pairing, so I did it properly: per-question differences between two configurations, Wilcoxon signed-rank, each question acting as its own control. Nothing reached significance. For a while I read that as "no difference."

It wasn't. Wilcoxon discards tied pairs, and with n non-zero pairs the smallest two-sided p-value it can return is 2^(1−n):

Non-zero pairs	Smallest achievable p
1	1.000
2	0.500
3	0.250
4	0.125
5	0.0625
6	0.031

Below six non-zero pairs, significance at α = 0.05 is arithmetically impossible regardless of the data.

My context-precision comparison had three non-zero pairs and returned p = 0.250. My recall comparison had one and returned p = 1.000. Both are exactly the floor — every non-zero pair pointed the same way, the most extreme result obtainable, and it still couldn't clear the threshold.

My sample collapsed twice over: 18 questions, then 8–9 matched pairs after judge failures, then 1–3 non-zero pairs after ties. Each stage looked survivable. Together they left a test with no power at all.

The tie rate is the more interesting half, though. Eight of nine pairs scored identically between k=8 and k=12. On most questions the chunks ranked 9 through 12 changed nothing the judge could see. That's a finding about retrieval, and it explains the null better than low power does.

The first thing my harness found was a bug in my own corpus

Before any of this, the eval surfaced something I hadn't gone looking for. My "21-paper corpus" was 15 papers. Each one's PDF and its metadata record were being indexed as separate works, so retrieving both counted as retrieving two relevant documents. Every precision number I'd computed was inflated.

Deduplicating on a canonical work ID fixed it. But I'd been quoting 21 for weeks, and nothing except an evaluation harness would have caught it.

What I'd do differently, and what I'd keep

Replicate before believing. At minimum three runs per configuration. If within-configuration spread approaches the between-configuration differences, report that and stop — it's the finding.

Report the judge and generator with every metric. A RAGAS score without them isn't comparable to anyone else's, or to your own from last month.

Check the power before spending the quota. Computing 2^(1−n) is one line and it tells you in advance whether a comparison can possibly reach significance. Mine couldn't, and I found out afterwards.

Report ties and sample attrition. "Not significant" and "no measurable difference" are different claims, and the gap between them lives in the counts that usually go unreported.

Classify the missing values, don't just count them. A missing score can be a generation failure, a judge parse failure, a metric undefined by design, or a genuine zero. Pooled together, a parser fix changes your reported mean by changing which questions are in it. [Suggested by Ahmet Özel (@ahmetozel) in the comments.]

Separate what a judge grades from what code can check. My suite has two tracks. Queries about corpus statistics and listings return zero chunks by design, so context-based metrics are undefined for them; those get deterministic assertions instead. Those assertions, and the intent-routing accuracy, came back at 100% across every run. Code-driven checks carried no variance at all. The LLM-judged metrics carried all of it. That contrast is worth designing for.

Limits

The replication spread conflates two sources of variance. My three identical-configuration runs re-generated answers and re-scored them, so the 0.22 recall range mixes generator sampling with judge variance, and I reported it as a single quantity. Separating them needs two runs I didn't do: replay the judge over saved answers to isolate judge variance, and re-generate under a fixed judge to isolate the other half. The harness can't do the replay yet, because it saves the answers but not the retrieved contexts the judge needs. Credit to Ahmet Özel (@ahmetozel) for pointing this out in the comments.

Run 1 at k=8 was scored on half the questions. The judge returned no score for 9 of the 18, and the nine it did score are the easier half of the set. So the run 1 figures in the table above, including the 0.833 recall, were compared against runs scored on all or nearly all of the questions. Restricting every run to the same nine questions, recall is 0.67 at k=3, 0.78 at k=5, 0.72 to 0.83 across the three k=8 runs, and 0.72 at k=12. There is no peak at k=8, and the spread within k=8 is 0.11, not 0.22, the same size as the spread between configurations. The conclusion stands, but part of the original inverted U came from comparing different question sets, not only from run-to-run noise.

This is one system, one 15-paper corpus, 18 scored content questions, and two judges. I'm not claiming a general law about RAGAS or about LLM-as-judge evaluation — I'm claiming that the standard procedure, run on a realistic small setup, produced a confident answer that replication dissolved, and that the failure modes were all invisible from the reported numbers.

If your evaluation is larger, it may be fine. If it's about this size, and you ran each configuration once, you have the result I had before I ran it again.

Harness, golden set, ablation and results: github.com/huali10044/research-rag. The system itself is live at huali-research-rag.streamlit.app.