Building an Evaluation Harness for Financial RAG: What I Learned About LLM-as-Judge Calibration

João Paulo Traguetta Rufino — Tue, 19 May 2026 22:12:31 +0000

I built a RAG system for financial document Q&A. It answers questions about SEC filings (revenue, margins, debt ratios) using 84 public company documents from the FinanceBench benchmark.

After running 100 queries, my LLM judge said 74% of answers were correct. The actual number was 27%.

This post is about how I found that gap, why it exists, and what I did about it.

The setup

The pipeline is straightforward: embed 84 SEC filings (10-K, 10-Q, earnings reports) into Qdrant with text-embedding-3-small, retrieve top-6 chunks per query, generate answers with GPT-4o-mini.

FinanceBench gives you 150 expert-annotated Q&A pairs with ground truth answers and source documents. I used 100 of them as my eval set.

I measured quality in two tiers:

Tier 1 — Retrieval. Did the system find the right document? I tracked Recall@6, Precision@6, and MRR.

Tier 2 — Generation. Is the answer any good? I used an LLM judge (GPT-4o-mini scoring 1-5) to evaluate Context Relevance, Answer Faithfulness, and Answer Relevance.

Retrieval: decent but not great

Metric	Value
Recall@6	0.830
Precision@6	0.422
MRR	0.646

83 out of 100 queries retrieved the correct source document. Not bad for vanilla semantic search with zero filtering.

The 17 misses were concentrated: Johnson & Johnson (9 misses across different doc types) and Adobe (5 misses). Together, 14 out of 17 failures came from just two companies.

Why? SEC filings use nearly identical language across companies. "Net revenues increased," "operating income was impacted by" — these phrases appear in every single 10-K. Embeddings can't reliably tell 3M's filing from Coca-Cola's when the language is this similar.

I confirmed metadata filtering fixes this. When I manually filtered Qdrant to only return chunks from the correct PDF, retrieval hit 100%. Automatic filtering (LLM extracts company from query, filters before retrieval) is the planned fix.

The judge lies

Here's where things got interesting.

Metric	Avg Score (1-5)
Context Relevance (C\|Q)	3.04
Answer Faithfulness (A\|C)	3.36
Answer Relevance (A\|Q)	3.96

The Answer Relevance judge classified 74 out of 100 answers as correct (score >= 4).

That felt too good for a system I knew was struggling. So I calibrated.

Calibration: the part nobody does

I took 30 query-answer pairs and manually compared them against FinanceBench's ground truth. My human accuracy was 27% — only 8 out of 30 were actually correct.

Then I checked the judge against my labels:

Metric	Value
TPR (sensitivity)	1.00
TNR (specificity)	0.55

TPR 1.00 means when an answer is correct, the judge always catches it. Good.

TNR 0.55 means when an answer is wrong, the judge only catches it 55% of the time. Almost half of wrong answers pass as correct.

Real example: the judge gave 5/5 to an answer saying "$1,608M" when the ground truth was "$2,018M." The response was well-structured, cited a source, used proper financial language. It just had the wrong number.

This is the core problem: the judge evaluates fluency, not factual accuracy. It can't verify numbers because it doesn't have the ground truth to compare against.

The fix: give the judge the answer key

I added a fourth metric — Answer Correctness (A|GT) — where the judge prompt includes the expected answer from FinanceBench alongside the model's response. Now the judge can actually check if "$1,608M" matches "$2,018M."

After adding A|GT:

Metric	Value
TPR	1.00
TNR	0.86

TNR went from 0.55 to 0.86. The judge now catches 86% of wrong answers.

With this calibrated judge, 53 out of 100 answers were correct. Not 74.

Two judges, two purposes

This isn't about one being better. They measure different things.

A|Q (no ground truth) simulates production. In a live system, you don't have the right answer — that's why the user is asking. This judge tells you if the response is coherent and relevant. Good for monitoring.

A|GT (with ground truth) is for development. When you have labeled data, you use it. This tells you if your pipeline is actually improving or if you're just getting more fluent wrong answers.

The mistake is using only A|Q during development and trusting the numbers. My pipeline looked like 74%. It was 53%.

What didn't work

Automatic metadata filtering via exact match. I tried extracting the company name with the LLM and filtering Qdrant by source filename. Problem: Qdrant's match filter does exact string matching, and "Johnson & Johnson" doesn't match JOHNSON_JOHNSON_2022_10K.pdf. Needs fuzzy or substring matching. Deferred to next phase.

Framework default judge prompts. Most RAG eval tools ship generic prompts that work for "does this make sense?" but fail for "is this number right?" If your domain requires factual precision, you need custom prompts and you need to calibrate them against human labels. There's no shortcut here.

Where things stand

Metric	Value
Retrieval Recall@6	0.830
Accuracy (calibrated)	53/100
Judge TPR	1.00
Judge TNR	0.86

The pipeline retrieves the right document 83% of the time but only gives the correct answer 53% of the time. The gap comes from retrieval misses (17%) and generation errors on correctly retrieved documents.

Next: systematic error analysis. Categorize every failure, pick the top 2 modes, fix them, measure impact.

Repo: financebench-rag-eval

References

FinanceBench — Patronus AI
6 RAG Evals — Jason Liu
LLM Evals FAQ — Hamel Husain

DEV Community: João Paulo Traguetta Rufino