Measuring RAG Quality With Hit Rate and MRR — LLM Zoomcamp Module 4

#llm #rag #python #ai

Most RAG systems ship without a single metric. Module 4 of LLM Zoomcamp fixes that.

Here's what I built and what the numbers showed.

The Setup

Knowledge base: 72 course lesson pages pulled from GitHub at a fixed commit so everyone works with the same data.

Ground truth: 360 questions generated by an LLM — 5 per page, worded differently from the source so keyword overlap doesn't inflate scores.

Three search methods under test: keyword search with minsearch, vector search with ONNX embeddings, and hybrid search combining both via Reciprocal Rank Fusion.

The Metrics

Hit Rate — did the right page appear in the top 5 results? Binary per question, averaged across 360.

MRR — how high up was the right page? First place = 1.0, second = 0.5, third = 0.33. Penalizes systems that find the answer but bury it.

The Numbers

Text search: Hit Rate 0.76, MRR 0.59

Vector search: Hit Rate 0.73, MRR 0.55

Hybrid search (k=1): Hit Rate 0.84, MRR 0.65

Text beat vector. Hybrid beat both. Tuning k from 60 to 1 in RRF added measurable MRR improvement.

The Takeaway

# benchmark any search function in one call
results = evaluate(text_search, ground_truth)
results = evaluate(vector_search, ground_truth)
results = evaluate(hybrid_search, ground_truth)

Once you have a ground truth dataset and an evaluate() function, every search decision becomes quantitative. Change a parameter, re-run, see the delta. No more guessing.