I built a small retrieval-augmented generation system over a corpus of research papers and wanted to answer an ordinary question: how many chunks s...
For further actions, you may consider blocking this person and/or reporting abuse
The attrition counts make the judge comparison more revealing than the headline scores. I would keep a fixed per-question table across every configuration, with separate fields for generation failure, judge parse failure, undefined metric and a valid zero score. Otherwise a parser improvement can change the average by changing its population.
For the repeated top-k sweep, saving each generated answer and rescoring it separately would help partition generator variance from judge variance. Then the same-answer judge replay and the full pipeline rerun answer different questions, while both preserve the question pairing and show which additional chunks were actually available.
Both of these are better than what I had, and the second one found a confound I'd missed.
On the per-question table — agreed, and the parser point is the part I hadn't thought through. Right now a missing score could be a generation failure, a judge parse failure, a metric that's undefined by design, or a genuine zero, and they're all pooled. So improving the parser would move my reported mean without anything about the system changing. I've added that to the checklist in the post.
On the sweep — you're right that my three identical-config runs re-generated and re-judged, so the 0.22 recall spread I reported mixes generator sampling with judge variance and I presented it as one number. The harness does store each generated answer, so replaying the judge over saved answers isolates the judge half without burning generation quota, and a re-generation under a fixed judge isolates the other. That's running next, and I'll post the split if it's interesting. I've added the caveat to the Limits section and credited you in both places.
Thanks — this was more useful than the post.
The added caveat separates what the original 0.22 spread can and cannot establish. For the saved-answer replay, I would store the judge configuration and parser version together with the answer identity, so the variance split remains interpretable after a parsing fix.
The regenerated-answer branch also benefits from using the same question set and judge settings as the replay branch. Reporting those paired per-question results beside each branch's unavailable-score counts would make it possible to see whether variation comes from a few unstable questions or a broad shift across the set.
Agreed on both, and checking the harness against your first point turned up more than I expected. It saves answers but not the retrieved contexts, the judge model or the RAGAS version, so the replay I described needs those stored first. I’m adding them, along with the reason each score is unavailable.
Your earlier missing-value point also applies to my headline number. Run 1 at k=8 was scored on 9 of 18 questions, and they were the first nine in order, which turn out to be the easier half. So the 0.83 that made k=8 look like a peak was compared against other values of k scored on the full set. Restricting every run to those same nine questions, recall is 0.67 at k=3, 0.78 at k=5, 0.72 to 0.83 across the three k=8 runs, and 0.72 at k=12. There is no peak, and the k=8 spread on matched questions is 0.11, not 0.22. Three of the nine questions account for all of the movement.
So the conclusion holds, but for a plainer reason than the one I gave: the inverted U was mostly a comparison between question sets. I’ve added a note to Limits. The regeneration branch will use the same question set and judge settings as the replay branch, reported per question with unavailable-score counts for each.
The matched nine-question comparison changes the interpretation substantially. I agree that the original peak was not evidence for an optimal k when its denominator was the easier half of the set. The 0.11 spread and three moving questions are the numbers I would keep next to the correction.
For the next run, freezing an explicit question-ID list before scoring would prevent this mismatch from recurring. Showing all eighteen rows with unavailable reasons would also keep the complete-set result distinct from a smaller matched subset.
Thank you, both suggestions are in the follow-up I just published: [dev.to/hualiphd/judge-noise-or-ans...]
On the three moving questions: on the nine matched questions, recall changed in only three of them (cont-01, cont-03, meta-05). The other six scored the same in all seven runs. One detail I hadn’t noticed: k=3’s entire deficit is a single question (cont-01), which k=3 missed and every other run found. So “k=3 under-retrieves” is a hint, not a finding.
On freezing the question list and showing every row: each run now records its question IDs and the golden-set hash, and every missing score carries a reason. The follow-up has an appendix listing all 18 questions across three generations, with the one missing score marked by its cause. Making the harness refuse to compare runs whose lists differ is my next change.
The follow-up also answers your first question, judge vs. generator variance. For faithfulness about 85% of the run-to-run variance comes from the answers. Recall and precision never read the generated answer, so with my deterministic retrieval all of their variance is the judge. That gave me measured noise bars, and the original sweep sits entirely inside them.
Top-k instability is the silent killer in production RAG. Seen too many teams "optimize" on a static eval set, ship it, then watch quality drift when the corpus grows or the embedding model gets swapped.
A few things that actually help:
1. Treat retrieval as a distribution, not a point estimate. Run your eval suite 10-20 times with different seeds (or just re-embed the corpus). If top-3 @ k=5 swings between 0.62 and 0.78 nDCG, your "optimal k" is a mirage. Report confidence intervals, not single numbers.
2. Decouple retrieval quality from generation quality. Build a small labeled set (50-100 queries) with ground-truth relevant doc IDs. Measure recall@k, MRR, nDCG independently of the LLM. If recall@10 plateaus at k=8 but your generator degrades after k=5 due to context pollution, that's a different problem — solve it with reranking or better chunking, not by pretending k=5 is "optimal."
3. Watch for semantic drift in the corpus. Adding 2k new papers can shift the embedding space enough that yesterday's k=7 becomes tomorrow's k=12. Schedule a monthly "retrieval regression" job: run your golden query set, alert if recall@k drops >5%.
4. Hybrid search > pure vector for research papers. BM25 on titles/abstracts + vector on full text, fused with RRF. The keyword signal catches exact terminology (method names, dataset IDs) that embeddings blur. Usually lets you drop k by 30-40% for same recall.
**5. If you're not logging the actual retrieved doc IDs per (site: labagent .tech)
Thanks, this is useful. A few of these lined up with what I found when I measured it: [dev.to/hualiphd/judge-noise-or-ans...]
On decoupling retrieval from generation: this turned out to be the key point for my setup. The RAGAS recall and precision metrics don’t read the generated answer at all; they compare retrieved chunks with the reference. My retrieval is deterministic, so all of their run-to-run variation came from the LLM judge. Counting-based recall@k against labeled document IDs would remove that noise entirely, so labeling the 18 questions with their relevant papers is next.
On logging retrieved doc IDs: fair hit. My harness saves the retrieved text for every question but not the document IDs. Adding that.
On distributions vs. point estimates: agreed. I didn’t get to 10–20 runs on a free-tier judge, but three to four runs per metric were enough to estimate how big a difference has to be before it means anything (about 0.09 for recall on 18 questions). Every gap in my original sweep falls under that.
Drift and hybrid search are good production points. My corpus is a fixed 15 papers, so drift doesn’t apply yet, and hybrid retrieval is on my list but untested.