I built a small retrieval-augmented generation system over a corpus of research papers and wanted to answer an ordinary question: how many chunks should I retrieve? So I did what the tutorials suggest — swept top-k across 3, 5, 8 and 12, scored each configuration with RAGAS, and read off the winner.
The answer was clean. k=8 led on all four metrics. Context recall was 0.53 at k=3, peaked at 0.83 at k=8, and fell back to 0.64 at k=12 — a textbook inverted U, with a story to match. Too few chunks and the generator starves; too many and distractors dilute the prompt. I had a number, a mechanism, and a plot.
Then I ran k=8 twice more, changing nothing.
| Run | Faithfulness | Answer relevancy | Context precision | Context recall |
|---|---|---|---|---|
| k=8, run 1 | 0.767 | 0.900 | 0.655 | 0.833 |
| k=8, run 2 | 0.590 | 0.804 | 0.585 | 0.611 |
| k=8, run 3 | 0.640 | 0.835 | 0.687 | 0.625 |
Recall ranged from 0.611 to 0.833 across three runs of an identical configuration. The 0.833 that made k=8 look like a peak was the high draw of three.
Set the variation within one configuration against the differences between configurations:
| Metric | Spread between configs (means) | Spread within k=8 | |
|---|---|---|---|
| faithfulness | 0.084 | 0.177 | noise exceeds signal |
| answer relevancy | 0.059 | 0.096 | noise exceeds signal |
| context precision | 0.023 | 0.102 | noise exceeds signal |
| context recall | 0.162 | 0.222 | noise exceeds signal |
On every metric, one configuration varies more against itself than the configurations vary against each other. k=5, k=8 and k=12 are not distinguishable on this evidence. The only ordering that survives is that k=3 under-retrieves — it has the lowest recall in every run.
The inverted U was an artifact of running each configuration once.
The judge moves the numbers more than the parameter does
While building the harness I had changed judge models — the free tier I started on ran out of quota. That turned out to be a second experiment I hadn't meant to run.
Same corpus, same generator, same k=5. Only the judge differs:
| Metric | Judge A | Judge B | Difference |
|---|---|---|---|
| faithfulness | 0.840 | 0.582 | 0.258 |
| context recall | 0.917 | 0.604 | 0.313 |
| context precision | 0.752 | 0.628 | 0.125 |
| answer relevancy | 0.777 | 0.861 | 0.084 |
Changing the judge moved faithfulness by 0.26 and recall by 0.31. Changing top-k from 3 to 8 — the parameter I was actually studying — moved faithfulness by 0.12.
Part of that gap is different grading. Part is something subtler: the two judges fail differently. One of them intermittently returned output the parser couldn't read, so those samples scored as missing and dropped out of the average. The two numbers were therefore computed over different subsets of questions. Both effects are properties of the judge, and both are invisible in a reported score.
I want to be careful here: I have one run per judge, so I can't cleanly separate a judge effect from an unlucky draw. The honest statement is that the judge shift is about the same size as the run-to-run variation I measured, which is already enough to make absolute RAGAS numbers meaningless without naming the judge that produced them.
The significance test couldn't have found anything
Comparing configuration means throws away the pairing, so I did it properly: per-question differences between two configurations, Wilcoxon signed-rank, each question acting as its own control. Nothing reached significance. For a while I read that as "no difference."
It wasn't. Wilcoxon discards tied pairs, and with n non-zero pairs the smallest two-sided p-value it can return is 2^(1−n):
| Non-zero pairs | Smallest achievable p |
|---|---|
| 1 | 1.000 |
| 2 | 0.500 |
| 3 | 0.250 |
| 4 | 0.125 |
| 5 | 0.0625 |
| 6 | 0.031 |
Below six non-zero pairs, significance at α = 0.05 is arithmetically impossible regardless of the data.
My context-precision comparison had three non-zero pairs and returned p = 0.250. My recall comparison had one and returned p = 1.000. Both are exactly the floor — every non-zero pair pointed the same way, the most extreme result obtainable, and it still couldn't clear the threshold.
My sample collapsed twice over: 18 questions, then 8–9 matched pairs after judge failures, then 1–3 non-zero pairs after ties. Each stage looked survivable. Together they left a test with no power at all.
The tie rate is the more interesting half, though. Eight of nine pairs scored identically between k=8 and k=12. On most questions the chunks ranked 9 through 12 changed nothing the judge could see. That's a finding about retrieval, and it explains the null better than low power does.
The first thing my harness found was a bug in my own corpus
Before any of this, the eval surfaced something I hadn't gone looking for. My "21-paper corpus" was 15 papers. Each one's PDF and its metadata record were being indexed as separate works, so retrieving both counted as retrieving two relevant documents. Every precision number I'd computed was inflated.
Deduplicating on a canonical work ID fixed it. But I'd been quoting 21 for weeks, and nothing except an evaluation harness would have caught it.
What I'd do differently, and what I'd keep
Replicate before believing. At minimum three runs per configuration. If within-configuration spread approaches the between-configuration differences, report that and stop — it's the finding.
Report the judge and generator with every metric. A RAGAS score without them isn't comparable to anyone else's, or to your own from last month.
Check the power before spending the quota. Computing 2^(1−n) is one line and it tells you in advance whether a comparison can possibly reach significance. Mine couldn't, and I found out afterwards.
Report ties and sample attrition. "Not significant" and "no measurable difference" are different claims, and the gap between them lives in the counts that usually go unreported.
Classify the missing values, don't just count them. A missing score can be a generation failure, a judge parse failure, a metric undefined by design, or a genuine zero. Pooled together, a parser fix changes your reported mean by changing which questions are in it. [Suggested by Ahmet Özel (@ahmetozel) in the comments.]
Separate what a judge grades from what code can check. My suite has two tracks. Queries about corpus statistics and listings return zero chunks by design, so context-based metrics are undefined for them; those get deterministic assertions instead. Those assertions, and the intent-routing accuracy, came back at 100% across every run. Code-driven checks carried no variance at all. The LLM-judged metrics carried all of it. That contrast is worth designing for.
Limits
The replication spread conflates two sources of variance. My three identical-configuration runs re-generated answers and re-scored them, so the 0.22 recall range mixes generator sampling with judge variance, and I reported it as a single quantity. Separating them needs two runs I didn't do: replay the judge over saved answers to isolate judge variance, and re-generate under a fixed judge to isolate the other half. The harness can't do the replay yet, because it saves the answers but not the retrieved contexts the judge needs. Credit to Ahmet Özel (@ahmetozel) for pointing this out in the comments.
Run 1 at k=8 was scored on half the questions. The judge returned no score for 9 of the 18, and the nine it did score are the easier half of the set. So the run 1 figures in the table above, including the 0.833 recall, were compared against runs scored on all or nearly all of the questions. Restricting every run to the same nine questions, recall is 0.67 at k=3, 0.78 at k=5, 0.72 to 0.83 across the three k=8 runs, and 0.72 at k=12. There is no peak at k=8, and the spread within k=8 is 0.11, not 0.22, the same size as the spread between configurations. The conclusion stands, but part of the original inverted U came from comparing different question sets, not only from run-to-run noise.
This is one system, one 15-paper corpus, 18 scored content questions, and two judges. I'm not claiming a general law about RAGAS or about LLM-as-judge evaluation — I'm claiming that the standard procedure, run on a realistic small setup, produced a confident answer that replication dissolved, and that the failure modes were all invisible from the reported numbers.
If your evaluation is larger, it may be fine. If it's about this size, and you ran each configuration once, you have the result I had before I ran it again.
Harness, golden set, ablation and results: github.com/huali10044/research-rag. The system itself is live at huali-research-rag.streamlit.app.
Top comments (10)
Top-k instability is the silent killer in production RAG. Seen too many teams "optimize" on a static eval set, ship it, then watch quality drift when the corpus grows or the embedding model gets swapped.
A few things that actually help:
1. Treat retrieval as a distribution, not a point estimate. Run your eval suite 10-20 times with different seeds (or just re-embed the corpus). If top-3 @ k=5 swings between 0.62 and 0.78 nDCG, your "optimal k" is a mirage. Report confidence intervals, not single numbers.
2. Decouple retrieval quality from generation quality. Build a small labeled set (50-100 queries) with ground-truth relevant doc IDs. Measure recall@k, MRR, nDCG independently of the LLM. If recall@10 plateaus at k=8 but your generator degrades after k=5 due to context pollution, that's a different problem — solve it with reranking or better chunking, not by pretending k=5 is "optimal."
3. Watch for semantic drift in the corpus. Adding 2k new papers can shift the embedding space enough that yesterday's k=7 becomes tomorrow's k=12. Schedule a monthly "retrieval regression" job: run your golden query set, alert if recall@k drops >5%.
4. Hybrid search > pure vector for research papers. BM25 on titles/abstracts + vector on full text, fused with RRF. The keyword signal catches exact terminology (method names, dataset IDs) that embeddings blur. Usually lets you drop k by 30-40% for same recall.
**5. If you're not logging the actual retrieved doc IDs per (site: labagent .tech)
Thanks, this is useful. A few of these lined up with what I found when I measured it: [dev.to/hualiphd/judge-noise-or-ans...]
On decoupling retrieval from generation: this turned out to be the key point for my setup. The RAGAS recall and precision metrics don’t read the generated answer at all; they compare retrieved chunks with the reference. My retrieval is deterministic, so all of their run-to-run variation came from the LLM judge. Counting-based recall@k against labeled document IDs would remove that noise entirely, so labeling the 18 questions with their relevant papers is next.
On logging retrieved doc IDs: fair hit. My harness saves the retrieved text for every question but not the document IDs. Adding that.
On distributions vs. point estimates: agreed. I didn’t get to 10–20 runs on a free-tier judge, but three to four runs per metric were enough to estimate how big a difference has to be before it means anything (about 0.09 for recall on 18 questions). Every gap in my original sweep falls under that.
Drift and hybrid search are good production points. My corpus is a fixed 15 papers, so drift doesn’t apply yet, and hybrid retrieval is on my list but untested.
The attrition counts make the judge comparison more revealing than the headline scores. I would keep a fixed per-question table across every configuration, with separate fields for generation failure, judge parse failure, undefined metric and a valid zero score. Otherwise a parser improvement can change the average by changing its population.
For the repeated top-k sweep, saving each generated answer and rescoring it separately would help partition generator variance from judge variance. Then the same-answer judge replay and the full pipeline rerun answer different questions, while both preserve the question pairing and show which additional chunks were actually available.
Both of these are better than what I had, and the second one found a confound I'd missed.
On the per-question table — agreed, and the parser point is the part I hadn't thought through. Right now a missing score could be a generation failure, a judge parse failure, a metric that's undefined by design, or a genuine zero, and they're all pooled. So improving the parser would move my reported mean without anything about the system changing. I've added that to the checklist in the post.
On the sweep — you're right that my three identical-config runs re-generated and re-judged, so the 0.22 recall spread I reported mixes generator sampling with judge variance and I presented it as one number. The harness does store each generated answer, so replaying the judge over saved answers isolates the judge half without burning generation quota, and a re-generation under a fixed judge isolates the other. That's running next, and I'll post the split if it's interesting. I've added the caveat to the Limits section and credited you in both places.
Thanks — this was more useful than the post.
The added caveat separates what the original 0.22 spread can and cannot establish. For the saved-answer replay, I would store the judge configuration and parser version together with the answer identity, so the variance split remains interpretable after a parsing fix.
The regenerated-answer branch also benefits from using the same question set and judge settings as the replay branch. Reporting those paired per-question results beside each branch's unavailable-score counts would make it possible to see whether variation comes from a few unstable questions or a broad shift across the set.
Agreed on both, and checking the harness against your first point turned up more than I expected. It saves answers but not the retrieved contexts, the judge model or the RAGAS version, so the replay I described needs those stored first. I’m adding them, along with the reason each score is unavailable.
Your earlier missing-value point also applies to my headline number. Run 1 at k=8 was scored on 9 of 18 questions, and they were the first nine in order, which turn out to be the easier half. So the 0.83 that made k=8 look like a peak was compared against other values of k scored on the full set. Restricting every run to those same nine questions, recall is 0.67 at k=3, 0.78 at k=5, 0.72 to 0.83 across the three k=8 runs, and 0.72 at k=12. There is no peak, and the k=8 spread on matched questions is 0.11, not 0.22. Three of the nine questions account for all of the movement.
So the conclusion holds, but for a plainer reason than the one I gave: the inverted U was mostly a comparison between question sets. I’ve added a note to Limits. The regeneration branch will use the same question set and judge settings as the replay branch, reported per question with unavailable-score counts for each.
The matched nine-question comparison changes the interpretation substantially. I agree that the original peak was not evidence for an optimal k when its denominator was the easier half of the set. The 0.11 spread and three moving questions are the numbers I would keep next to the correction.
For the next run, freezing an explicit question-ID list before scoring would prevent this mismatch from recurring. Showing all eighteen rows with unavailable reasons would also keep the complete-set result distinct from a smaller matched subset.
Thank you, both suggestions are in the follow-up I just published: [dev.to/hualiphd/judge-noise-or-ans...]
On the three moving questions: on the nine matched questions, recall changed in only three of them (cont-01, cont-03, meta-05). The other six scored the same in all seven runs. One detail I hadn’t noticed: k=3’s entire deficit is a single question (cont-01), which k=3 missed and every other run found. So “k=3 under-retrieves” is a hint, not a finding.
On freezing the question list and showing every row: each run now records its question IDs and the golden-set hash, and every missing score carries a reason. The follow-up has an appendix listing all 18 questions across three generations, with the one missing score marked by its cause. Making the harness refuse to compare runs whose lists differ is my next change.
The follow-up also answers your first question, judge vs. generator variance. For faithfulness about 85% of the run-to-run variance comes from the answers. Recall and precision never read the generated answer, so with my deterministic retrieval all of their variance is the judge. That gave me measured noise bars, and the original sweep sits entirely inside them.
The single cont-01 miss is a much narrower result than a general claim that k=3 under-retrieves. Keeping that question visible also makes it possible to distinguish a retrieval boundary case from a broad setting effect.
For the next comparison, I would repeat each k against the same saved retrieval outputs as well as fresh answers. That separates the uncertainty in the score difference from the variance of either score alone. The 85% answer contribution is useful for this setup, but it should travel with the model, prompt and question cohort that produced it.
Agreed, cont-01 reads as a retrieval boundary case, and keeping it visible is what lets you tell that apart from a setting effect.
On the paired design: the bars in the follow-up assume the two runs’ noise is independent, so the noise of a difference comes out at about 1.4 times a single run’s. Comparing two k values on the same questions with the same judge, per-question noise is probably correlated, so measuring the spread of the difference directly should give a tighter and more honest bar. That’s how I’ll run the next sweep: re-judge each k’s saved retrievals several times for recall and precision, and regenerate answers for each k for faithfulness and relevancy, then compare the paired differences.
And fair point on the 85%: it’s a property of this generator, prompt, judge and question set, not a general number. I’ll state it that way.