Judge noise or answer noise? What really moves RAG evaluation scores

Summary. RAG evaluation scores change from run to run even when nothing in the system changes. I measured where that variation comes from in my own system: from the LLM judge grading the same answers differently, or from the generator writing different answers. For faithfulness, about 85% of the run-to-run variance came from the answers. For answer relevancy, nearly all of it did. Context recall and context precision never read the generated answer, so with deterministic retrieval all of their variance came from the judge, even with the judge set to its most consistent setting. From these measurements, a difference between two settings, each run once on my 18 questions, needs to exceed about 0.15 (faithfulness), 0.08 (relevancy), 0.09 (recall) or 0.10 (precision) to mean anything. Applying that bar to the top-k sweep in my first post confirms that the sweep could not rank the k values, and that the "optimal k=8" came from uneven question coverage and single runs.

Figure 1. Gray dashed line: the recall curve published in the first post (one run per k, unequal question sets). Blue dots: every run of that sweep, scored on the 9 questions all seven runs scored. Shaded band: where a single run's 9-question recall lands from judge noise alone (mean ± 0.09).

Key findings

1. The judge does not always give the same verdict twice, even when set to be as consistent as possible. Grading the same answers three times moved the 18-question means by 0.003 (relevancy) to 0.085 (precision).
2. For faithfulness and relevancy, the answers matter more than the judge. Run-mean standard deviation rose from 0.021 to 0.054 for faithfulness and from 0.002 to 0.028 for relevancy when the answers were regenerated.
3. Recall and precision are pure judge noise in this setup. RAGAS computes them from the question, the retrieved chunks and the reference answer. Regenerating answers cannot change them.
4. Noise bars for comparing two single runs: about 0.15 faithfulness, 0.08 relevancy, 0.09 recall, 0.10 precision (18 questions; about 40% wider on 9).
5. The first post's k=8 peak does not survive. On the 9 questions every run scored, all seven sweep runs fall inside the judge-noise band (Figure 1). The first post's claim that changing the judge model moves scores more than changing k gets stronger.
6. Replicate the right thing. For recall and precision, re-judge saved results. For faithfulness and relevancy, regenerate answers. Re-judging alone would make faithfulness look more than twice as stable as it is. ## 1. Background

In the first post I swept top-k over a small RAG system, found a clean winner at k=8, then ran k=8 twice more and watched the winner dissolve: same configuration, recall anywhere from 0.61 to 0.83.

A commenter, Ahmet Özel, asked the obvious next question: how much of that spread is the judge, and how much is the generator writing different answers? The two need different fixes. If the judge is noisy, you grade more times. If the answers are noisy, you generate more times. I had been doing neither, and I couldn't tell which one I needed.

2. Setup

Same system as before: 15 research papers, 18 judged questions, k=8. The generator is gpt-oss-20b on Groq. The judge is Gemini 3.5 Flash Lite, through RAGAS 0.2.15 with four metrics: faithfulness, answer relevancy, context precision and context recall.

One setting matters for what follows. An LLM's temperature controls how much randomness goes into choosing each word: higher values give more varied output, and 0 tells the model to always pick its most likely next word, which in principle makes the same input produce the same output. My harness sets the judge's temperature to 0, the usual advice for LLM judges. The generator runs at the provider's default temperature, so its answers are expected to vary from run to run.

Two experiments:

1. Judge only. Generate one set of answers, save them with their retrieved chunks, and have the judge score the same saved answers three times. Nothing is regenerated or re-retrieved. The prompts sent to the judge had identical hashes in all three passes, so any difference is the judge.
2. Generator plus judge. Generate three separate sets of answers under identical settings and score each once. To make the first experiment possible, the harness had to learn to replay: save every answer and its retrieved chunks, then re-run only the judge. It now also records why any score is missing (rate limit, timeout, unparseable judge output and so on) and reports each mean with its coverage, like 0.833 (9/18). That second change is how I found out that the 0.833 in my first post covered only half the questions.

3. Results

3.1 The judge alone

Same answers, same chunks, same prompts, three passes:

Metric	Pass 1	Pass 2	Pass 3	Range
Faithfulness	0.653	0.611	0.628	0.042
Answer relevancy	0.821	0.824	0.824	0.003
Context precision	0.601	0.686	0.632	0.085
Context recall	0.694	0.648	0.667	0.046

Setting the judge's temperature to 0 did not make it repeat itself. Each metric varies for a reason specific to how it's built.

Faithfulness splits the answer into claims, then checks each claim against the chunks. The judge split identical answers into different numbers of claims: one answer came out as 5, 10 and 11 claims in the three passes. The score is supported claims over total claims, so the denominator moves. Faithfulness changed on 11 of 18 questions, and those were exactly the 11 where the claim count changed.

Precision and recall are built from yes/no verdicts. 5 of 148 chunk verdicts flipped between passes, and 3 of 22 reference-statement verdicts. That sounds small until you notice that 14 of my 18 reference answers were a single sentence. For those questions, recall is one yes/no call, so a single flip moves that question from 1 to 0. One question went 1, 0, 0 across three identical passes.

Relevancy barely moved. The judge writes questions the answer seems to address, and an embedding model compares them with the real question. The embedding step smooths out the judge's wording.

3.2 Adding new answers

Retrieval returned exactly the same 8 chunks for every question in all three generations, so the only thing that changed was the generated text, and it changed a lot: most answers to content questions shared less than a fifth of their wording with the previous generation.

Spread of the 18-question mean, as a standard deviation across runs:

Metric	Judge only	New answers, each judged once	Share from the answers
Faithfulness	0.021	0.054	about 85%
Answer relevancy	0.002	0.028	nearly all
Context recall	0.032	—	none
Context precision	0.035	—	none

The blanks are the most useful finding of the whole exercise. In RAGAS, context recall and context precision never read the generated answer. They compare the retrieved chunks with the question and the reference answer. My retrieval is deterministic, so regenerating answers cannot change those two scores, and all of their run-to-run noise is the judge.

I would have guessed the opposite split before running this. I'd assumed the judge was the main problem everywhere.

3.3 How big a difference needs to be

Compare two settings, say k=5 against k=8, each run once on these 18 questions. Each run's mean has its own noise, the difference between two runs has about 1.4 times that, and the difference should be about twice its own noise before you believe it:

Metric	Smallest difference worth reporting
Faithfulness	about 0.15
Answer relevancy	about 0.08
Context recall	about 0.09
Context precision	about 0.10

Each bar works like a 95% interval on the difference between two runs. They come from three or four runs per metric, so treat them as rough; the true values could plausibly be half or double. More runs would tighten them, but each full run costs about 257 judge calls against a free-tier limit of 500 a day. On a 9-question subset the bars are about 40% wider.

4. What this changes in my first post

Figure 1 puts the first post's sweep against the measured noise band. Going through that post's conclusions one at a time:

"k=8 is the optimum." Gone, and now I know why. Two things built that peak. First, the runs were scored on different questions: the k=8 run behind the 0.83 had only 9 of 18 questions scored, because the judge failed on the rest, and the questions it failed on were the harder ones. Second, each setting was run once. Put every run on the same 9 questions and the curve flattens to between 0.67 and 0.83. All seven runs fit inside the band, so nothing in the sweep can be told apart from noise. Only three of those nine questions changed score in any run (cont-01, cont-03 and meta-05); the other six scored the same in all seven.

"Noise exceeds signal on every metric." Still true, but I overstated the noise. The first post said recall varied by 0.22 across three k=8 runs. Part of that was the same coverage problem: on matched questions the k=8 runs spread by 0.11. The conclusion stands because the between-setting differences shrank too. What's new is knowing where the noise comes from: for recall and precision it's all the judge, and for faithfulness it's mostly the answers.

"k=3 under-retrieves." Weakened to a hint. k=3 still has the lowest recall on the matched questions (0.67, against about 0.77 for the other runs on average), and it sits on the lower edge of the band. That's consistent with retrieving too little, but the gap is below the 0.13 bar for 9 questions, and all of it comes from one question (cont-01), which k=3 missed and every other run found. I'd now call it a direction worth testing, not a finding.

"The judge moves the numbers more than the parameter does." Stronger. I hedged this last time because I had only one run per judge and couldn't separate a judge effect from an unlucky draw. Now I know the same judge's run-to-run noise: the bar is about 0.09 for recall and 0.15 for faithfulness. Switching judge models moved recall by 0.31 and faithfulness by 0.26: more than three times the bar for recall, and nearly twice the bar for faithfulness. One caveat from the first post still applies: the two judges failed on different questions, so part of that gap is coverage. But it is too large to be run-to-run noise.

"The significance test couldn't find anything." Now I know why so many pairs were tied. The first post blamed Wilcoxon's floor for dropping tied pairs: 8 of 9 pairs scored identically between k=8 and k=12. The underlying reason is the references. 14 of my 18 reference answers were a single sentence, so recall on those questions can only be 0 or 1. Any two settings that retrieve the same relevant chunk tie exactly, and the test has nothing left to work with.

"Replicate before believing." Still right, but replicate the right thing. For recall and precision, re-judge the same saved results; it's also cheaper, since there are no generation calls. For faithfulness and relevancy, regenerate the answers, because re-judging captures only about 15% of faithfulness's variance.

5. Failure cases the per-question view showed

Faithfulness handles a correct "no" badly. One question asks which paper won a best-paper award. None did, and all three generations said so correctly. Faithfulness scored them 1.00, 0.33 and 0.00. The "no" itself isn't checkable against the chunks, so the score depends on whatever incidental sentences surround it.

Relevancy rewards confidence, including invented confidence. One question asks which cognitive biases a paper addresses. The papers don't name specific biases. One generation said so and scored 0 on relevancy, because the metric treats a hedged answer as noncommittal. Another generation confidently listed "confirmation bias", which isn't in the corpus, and scored 0.58. Faithfulness caught the invention (0.17), which is the argument for never reading relevancy alone.

Missing scores have causes, and the causes are fixable. Across about 1,200 judge calls, two failed because the judge wrote malformed JSON escapes. On one day the judge slowed to about ten times its usual response time, and context precision timed out on 16 of 18 questions while the other three metrics finished. The cause: RAGAS's time limit covers a whole metric on one question, not each call, and precision makes one judge call per retrieved chunk (8 at k=8, against one to three for the others). A slow judge hits precision first, every time. Without per-reason logging, all of that would have shown up as NaN and silently shrunk the average.

6. Recommendations

• Freeze the question list before scoring, and report every row. Fix the list of question IDs before any run, compare runs only on that list, and show all of them, with the reason for each missing score, so a complete-set result is never confused with a smaller matched subset. Appendix A shows this for the three generations. (Suggested by Ahmet Özel in a follow-up comment.)
• Report coverage with every mean. 0.833 and 0.833 (9/18) are different claims.
• Where you can, score retrieval without an LLM. If each question is labeled with the documents that should be retrieved, recall@k and similar measures can be computed by simple counting. With deterministic retrieval they would have no run-to-run noise at all, which is exactly where the judge-based recall and precision are weakest here. (Suggested by another reader.)
• Know which metrics read the answer. Retrieval metrics need re-judging; answer metrics need regenerating.
• Compute a noise bar before comparing settings. Two or three runs per setting are enough to estimate one.
• Write references as several checkable claims. A one-sentence reference turns recall into a coin flip.
• Name the judge, its version and the library version next to every number. ## 7. Limits

One corpus of 15 papers, 18 questions, one value of k, one generator and one judge. The variance estimates come from three or four runs each. The September sweep used a Gemini judge whose exact model version wasn't recorded, so the noise band in Figure 1 is borrowed from October's judge and the re-reading is indicative, not exact. The thresholds are specific to this setup; the method for getting them is not.

8. Next

I've rewritten the 13 single-sentence references as two to four claims each. Re-judging the saved results against them will show whether multi-claim references reduce the judge noise on recall, and by how much. Next, I'll label each question with the papers that should be retrieved and log the retrieved document IDs on every run; my harness currently saves the retrieved text but not the IDs. That gives a judge-free retrieval score to set beside the judge-based one. After that, the top-k sweep again, with enough replicates per setting to clear the bar.

The harness, the result files and every raw judge output are in the repo: github.com/huali10044/research-rag.

Appendix A. All 18 questions, three generations

Faithfulness and answer relevancy for each generation, each judged once. Every question is listed; a missing score shows its recorded reason.

Question	Faithfulness G1	G2	G3	Relevancy G1	G2	G3
meta-01	1.00	0.83	1.00	0.94	0.97	— (judge output unreadable)
meta-02	0.75	1.00	1.00	0.97	0.98	0.98
meta-03	1.00	1.00	0.89	0.78	0.78	0.85
meta-04	0.33	1.00	1.00	0.95	0.94	0.96
meta-05	0.50	0.50	0.33	0.72	0.67	0.70
cont-01	1.00	0.60	0.90	0.97	0.78	0.70
cont-02	0.80	0.78	0.79	0.99	0.99	0.99
cont-03	0.47	0.23	0.33	0.83	0.90	0.99
cont-04	0.26	0.00	0.12	0.96	0.98	0.80
cont-05	0.80	0.17	0.73	0.00	0.58	0.87
cont-06	0.43	0.38	0.38	0.93	0.89	0.88
cont-07	0.61	0.25	0.33	0.73	0.81	0.78
cont-08	0.56	0.52	0.43	0.98	1.00	1.00
cont-09	0.80	1.00	1.00	0.71	0.67	0.61
cont-10	0.29	0.22	0.40	0.64	0.85	0.76
cont-11	0.56	0.28	0.45	0.70	0.87	0.58
adv-01	0.60	0.82	0.14	0.84	0.98	0.83
adv-02	1.00	0.33	0.00	0.95	0.95	0.87