DEV Community: elvisyao007

faithfulness spread = 0.000: what self-grading RAG eval actually looks like

elvisyao007 — Sun, 07 Jun 2026 18:22:53 +0000

description: "I ran my RAG eval twice — once with the same model grading itself, once with an independent judge from a different family. Here's what changed, and why spread = 0.000 is the tell."

Last post I claimed something specific: faithfulness scored 0.67, but an independent judge found 33 of 100 answers were grounded in context and still factually wrong.

A fair question: why trust that judge?

I have a concrete answer, because I ran the eval twice. The first run used the same model for both generation and judging — self-grading. The second run used a completely different model family as the judge. Here are the numbers from both.

The before and after

Metric	Self-judge (qwenj, same model)	Independent judge (gemma4:31b)
faithfulness mean	0.7751	0.6662
faithfulness spread	0.0000	0.0500
grounded-but-wrong	48 / 100	33 / 100

Read the spread row. The self-judge returned a spread of exactly 0.0000 — not "near zero," literally zero. Every query returned an identical faithfulness distribution. The judge was not reading the answers. It was rubber-stamping.

The independent judge returned a spread of 0.05. Small, but non-zero: the judge was actually discriminating between better and worse answers.

Everything else follows from that single difference.

Why spread = 0.000 is the tell

A judge that is genuinely evaluating will find some answers more faithful than others — it will disagree with itself across queries. A judge that has collapsed into rubber-stamping gives the same score to everything, because it has stopped reading. The variance goes flat.

Non-zero spread is necessary but not sufficient evidence of a good judge. A random judge also has spread. The spread check rules out the worst case — the complete collapse of judgment — not all cases. The gold standard is still human-label agreement on a sampled subset. But zero spread is an immediate red flag that something is wrong.

The self-judge gave faithfulness 0.7751. That number is almost certainly inflated. When the same model generates an answer and then evaluates it, it tends to recognize its own phrasing and reward it. The technical term is self-enhancement bias — a documented effect that scales with model capability and persists even when authorship is hidden.

What inflated faithfulness does downstream

Faithfulness inflation doesn't just change one number. It cascades.

The self-judge scored more answers as "faithful" (inflated 0.7751 vs 0.6662). A larger faithful pool means more opportunities to be grounded-but-wrong. That's why the self-judge found 48 grounded-but-wrong answers while the independent judge found 33: the self-judge was counting answers as "grounded" that the independent judge correctly did not. False positives in faithfulness create false positives in grounded-but-wrong.

The independent judge, being more accurate about faithfulness, shrank both numbers toward reality.

How I built the independent judge

Three things that matter:

Cross-family split. My generator is qwen3:32b (Qwen, Alibaba). My judge is gemma4:31b (Gemma, Google). Different model, different family, different training lineage. Self-preference bias leaks across a model family, not just an exact checkpoint — using a different Qwen checkpoint as the judge would still be suspect. The key is the family boundary.

Ground-truth anchor. Self-preference bites hardest on subjective tasks where there's no right answer to compare against. JQaRA ships gold answers. My correctness check asks the judge to compare the model's answer against the gold answer — not to issue a free-floating opinion. Anchoring on a reference shrinks the surface where bias can hide.

The on-prem cost. On a single RTX 5090 with 32 GB VRAM, qwen3:32b (20 GB) and gemma4:31b (19 GB) can't both be resident at the same time. I had to build a two-pass architecture: all generation first, then explicit VRAM unload, then all judging. This also required routing around the OpenAI-compat endpoint — thinking-capable models exhaust max_tokens with reasoning tokens before emitting content, so I used Ollama's native /api/chat with think=false. None of this is hard, but it's the operational reality of doing this properly on-prem, and it's the kind of friction that makes most people default to self-judging in a single pass.

Being honest about the limits

Non-zero spread rules out rubber-stamping. It doesn't prove the judge is calibrated. For that, you need to hand-label a sample — grade 30–50 answers yourself and measure how often the judge agrees. I haven't published that calibration for this run yet. The spread check is a fast sanity gate, not the finish line.

What to gate RAG eval on

An independent judge — different family, not just different checkpoint. Self-judging numbers are theater.
Ground truth where it exists. A reference answer reduces the bias surface more than any prompting trick.
Spread as a sanity check. Report it alongside the mean. Zero spread = stop, something is wrong.
Human-label calibration on a sample before you trust the judge in production.

The self-judging run gave a clean-looking 0.77 faithfulness with zero spread. The independent run gave 0.67 with 0.05 spread, and found 15 fewer grounded-but-wrong answers. The real system was worse than the self-judge claimed and better-characterized than the inflated number suggested. The 0.67 is more credible precisely because it's lower.

The full run — both phases, infrastructure fixes, raw scores — is here: github.com/elvisyao007/eval-driven-llm. Next I'm going after context_recall = 0.41 with hybrid retrieval, judged by the same independent setup. Following the build in public.

My RAG's faithfulness was 0.67. 1 in 3 answers were still wrong.

elvisyao007 — Sun, 07 Jun 2026 17:02:51 +0000

description: "An on-prem JQaRA eval. Reranking nudged P@1 but the system was still wrong a third of the time. Why faithfulness alone is a trap, and what to gate on instead."

I built a small Japanese RAG system, ran it entirely on my own hardware (RTX 5090, Ollama), and evaluated it with an independent judge model instead of letting the generator grade its own homework.

Two things surprised me, and they're connected:

Adding a reranker — the move everyone reaches for first — barely moved the needle.
My faithfulness score looked acceptable (0.67), yet 33 out of 100 answers were grounded in the retrieved context and still factually wrong.

This post is about why those two facts are the same story, and why a faithfulness gate alone would have shipped a system that's wrong a third of the time without ever flagging it.

TL;DR

Reranking improved P@1 by +1.3 points but lowered Recall@10. It reorders what retrieval already found; it can't retrieve what retrieval missed.
The real bottleneck was recall (context_recall = 0.41): the evidence needed to answer often wasn't retrieved at all.
faithfulness = 0.67 is a trap. Faithfulness measures whether an answer is consistent with the retrieved context — not whether it's correct. An answer grounded in wrong-but-retrieved context scores as faithful.
An independent correctness judge found 33/100 "grounded-but-wrong" answers — confidently wrong, fully grounded, invisible to faithfulness.
Lesson: faithfulness is necessary, not sufficient. Gate on answer-correctness + context_recall, and stop reaching for a reranker when recall is your problem.

The setup (so you can trust the numbers)

Component	Choice
Benchmark	JQaRA (じゃくら) — Japanese QA-for-retrieval, built on the JAQKET quiz set
Retrieval eval	1,667 queries, deterministic
Generation eval	100 queries
Generator	`qwen3:32b`
Judge	`gemma4:31b` — a different model from the generator
Hardware	single RTX 5090, on-prem, Ollama

The judge being a different model matters, and I'll come back to why.

Act 1: the obvious move — add a reranker

The standard RAG upgrade path: dense retrieval is your first stage, a cross-encoder reranker is your second. So I added one and re-ran retrieval.

Metric	Dense	Dense + rerank	Δ
P@1	0.8308	0.8440	+0.0132
Recall@10	0.5738	0.5634	−0.0104

Read that carefully. The reranker did exactly what a reranker does: it sharpened the top of the list (P@1 up — the single best document lands at rank 1 more often) while slightly demoting some relevant docs out of the top 10 (Recall@10 down). That's a precision-for-recall trade, not a free win.

And here's the thing that should give you pause: if your generator reads more than the top result — top-5, top-10 — that recall drop can hurt downstream answers even as P@1 improves. The metric you celebrate isn't the metric that feeds your generator.

The deeper problem: a reranker reorders the candidate set. It cannot conjure a document that dense retrieval never surfaced. Which brings us to the number that actually mattered.

Act 2: the metric I trusted too much

I moved to generation eval expecting faithfulness to be the headline. It came back at 0.6662. Mediocre, but the kind of number you squint at and think "okay-ish, ship the next iteration."

That instinct is the trap.

Metric	Value	What it actually tells you
faithfulness	0.6662	"Looks okay" — and is dangerously incomplete
faithfulness spread	0.0500	Non-zero → the judge is discriminating, not rubber-stamping
context_recall	0.4062	The real bottleneck — evidence often wasn't retrieved
grounded-but-wrong	33 / 100	The failures faithfulness structurally cannot see

Faithfulness measures consistency with the retrieved context, not correctness against ground truth. An answer that faithfully reports a wrong-but-retrieved passage is, by definition, faithful. So a grounded-but-wrong answer doesn't lower your faithfulness score — it sits in the "good" portion of it. Optimize for faithfulness and you are partly optimizing toward confident, well-grounded, wrong answers.

To catch this you need a separate question: is the answer actually correct? I ran that as an independent correctness check against JQaRA's gold answers. The essence:

# Not "is the answer supported by the context?" (faithfulness)
# But "is the answer correct vs the gold answer?" (correctness)

judge(question, model_answer, gold_answer) -> {correct | incorrect}
grounded_but_wrong = faithful(answer) AND NOT correct(answer)

Result: 33 of 100 answers were faithful and wrong at the same time. A faithfulness gate would have waved every one of them through.

Why this happened: recall was the leak

The three numbers line up into one causal chain:

context_recall = 0.41 → for most queries, the passage that actually answers the question wasn't in the retrieved context.
The generator answers anyway, grounding itself in whatever was retrieved — confidently, fluently.
That answer is faithful (grounded in retrieved text) and wrong (the retrieved text didn't contain the answer). → grounded-but-wrong.

So context_recall is the leading indicator, grounded-but-wrong is the lagging confirmation, and faithfulness is the misleading number in the middle that papers over both.

And now Act 1 and Act 2 close into the same loop: I reached for a reranker, but reranking optimizes the wrong stage when recall is your bottleneck. No amount of reordering fixes a document that was never retrieved. The right lever was upstream — chunking, embedding model, hybrid (lexical + dense) retrieval, query expansion — not a cross-encoder polishing a list that's missing the answer.

A note on judge independence (why the spread matters)

If you let a model grade its own outputs, it tends to like them — LLM-as-judge has a well-documented self-preference bias, and a self-judging setup often produces near-1.0 scores with almost no variance. That near-zero spread is the tell.

My judge (gemma4:31b) is a different model from the generator (qwen3:32b), and the faithfulness spread came back at 0.05 — non-zero. Small, but it's the proof that the judge is actually discriminating between good and bad answers rather than rubber-stamping. If you take one process habit from this post, take this one: never let the model that wrote the answer be the model that scores it.

What I'd actually gate a production RAG on

Most "RAG eval" stops at faithfulness because it's the easiest to compute. That's exactly why it's the wrong place to stop. The gate I'd ship behind:

Answer-correctness vs ground truth — the metric that actually catches grounded-but-wrong. Non-negotiable.
context_recall — your leading indicator. If this is low, fix retrieval before you touch the generator or reach for a reranker.
faithfulness — keep it, but only as a hallucination guard on top of correctness, never as a stand-in for it.
An independent judge — different model, and watch the score variance to confirm it isn't rubber-stamping.

A demo proves the happy path works. A system you'd put in front of a business has to know — and prove with numbers — how often it's confidently wrong. The gap between those two is exactly this eval discipline.

Code, the eval harness, and the raw run are here: github.com/elvisyao007/eval-driven-llm. Next I'm going after that context_recall = 0.41 — hybrid retrieval and chunking experiments, measured the same way. Following the build in public.

If you run RAG eval and only look at faithfulness, go check your grounded-but-wrong rate. I'd bet it's not zero.