DEV Community

Dicardo9
Dicardo9

Posted on

Practice RAG Retrieval Metrics Offline — A Tiny Stdlib Eval Loop (Synthetic Data)

#python #machinelearning #rag #llm

If you are learning RAG, you eventually hit the same wall:

“I can chat with my docs… but I have no idea if retrieval is actually improving.”

Most tutorials jump straight to embeddings, vector DBs, and LLM judges. That is fine for demos. It is a poor first step for understanding retrieval metrics, because too many moving parts hide what the numbers mean.

This post is a walkthrough of a deliberately tiny retrieval eval loop:

  • synthetic docs + QA gold labels
  • a toy lexical retriever (intentionally dumb)
  • precision@k / recall@k / hit@k printed to the terminal
  • Python stdlib-first — no model weights, no API keys

I shipped this as two small paid packs (Lite / Pro) on Payhip. Soft links are at the end. The article itself is meant to stand alone as an educational note.

Honest scope up front: this is a learning harness, not a production RAG stack. Scores on synthetic data are not business scores.

Why start with retrieval metrics (not the generator)?

RAG quality has at least two layers:

  1. Retrieval did we pull the right documents / chunks?
  2. Generation given that context, did the model answer faithfully?

People often jump to “faithfulness” or answer correctness” first. Those are valuable — and expensive. They need a generator, a judge prompt, and careful sampling.

Retrieval metrics are cheaper and clearer for beginners:

Metric Plain-English meaning
Precision@k Of the top-k retrieved items, what fraction are gold?
Recall@k Of all gold items, what fraction appear in top-k?
Hit@k Did top-k contain at least one gold item? (0/1 per query)

If Hit@k is low, your generator never had a fair chance. Fix retrieval first.

In a minimal kit, faithfulness is N/A: there is no generator. Do not invent a fake 0.95 to look complete.

The smallest useful eval loop

Conceptually:

docs.jsonl  ─►  toy retriever  ──►  top-k doc ids
qa.jsonl    ──►  gold_doc_ids   ──►  compare  ──►  P@k / R@k / Hit@k```

### Example data shapes

**Document** (one JSON object per line):



```json
{"doc_id": "d1", "title": "Leave policy", "text": "Annual leave is 15 days..."}
Enter fullscreen mode Exit fullscreen mode

QA gold (one JSON object per line):

{"qid": "q1", "question": "How many annual leave days?", "gold_doc_ids": ["d1"], "gold_answer": "15 days"}
Enter fullscreen mode Exit fullscreen mode

You can later swap in your own desensitized files with the same fields. Do not dump customer names, internal URLs, or secrets into a shareable folder.

Toy retriever (on purpose)

A lexical / overlap scorer is enough to practice the eval plumbing:

  • tokenize question + docs
  • score by overlap
  • return top-k ids
  • score against gold_doc_ids

It will look “too good” or “too brittle” on tiny synthetic sets. That is fine. The goal is to learn the metric definitions and failure modes, not to ship a search engine.

How to read the numbers without lying to yourself

Suppose k=3 and each question has exactly one gold doc.

  • Precision@3 often sits near 0.33 even when you “hit — because only 1 of 3 slots can be gold.
  • Recall@3 can look perfect (1.0) while the ranking is still noisy.
  • Hit@3 collapses to “did we get the right doc somewhere in top-3?” — useful, but easy questions make everyone score 1.0.

Common misreads:

  1. High synthetic scores ≠ production readiness. Tiny fake corpora overfit your toy retriever.
  2. Precision dropped while recall rose. You may be retrieving wider and noisier. Whether that is “better” depends on whether you fear misses more than hallucinations.
  3. Nothing moved when you toggled a knob. Your questions may not discriminate. Add harder / boundary cases before more tuning.
  4. Faithfulness filled in by hand. If you did not run a generator+judge, leave it N/A.

Rule of thumb for experiments: change one variable at a time, keep the same dataset and same k, then compare.

Chunking and rerank: when they matter (and when they don't)

Once the single-path loop makes sense, the next educational step is controlled contrasts:

Knob What you are testing Honest caveat
Chunk strategy (none / fixed / sliding) Does splitting long docs help or cut answers apart? More chunks ≠ better; can add noise
Toy rerank on/off Does reordering top candidates change Hit/P/R? A demo toggle is not a production cross-encoder

Heuristics (not promises):

  • Long docs, answers buried mid/late → try sliding / size first.
  • First-pass recall OK but users always click result #2 → consider a real reranker later.
  • Pretty metrics, ugly production usually a eval-set gap, not “need a bigger model.”

With few questions (e.g. <30), do not make product decisions on a 0.02 swing.

Why not just use Ragas / TruLens / ?

Those tools are excellent when you already know what you are measuring and can afford API/judge cost.

A tiny offline harness is for a different moment:

  • you want to feel P@k / R@k / Hit@k on a laptop
  • you do not want to configure a vector DB yet
  • you want an empty requirements.txt (stdlib-first) so the lesson is the metrics, not the dependency graph

When you outgrow it, replace the toy retriever with your real pipeline and keep the same gold JSONL + metric definitions. That migration path is the point.

What I built (and what I did not claim)

I packaged the above as:

  • RAG Eval Lite — one-shot loop: synthetic docs/QA → toy retrieve → print mean P@k / R@k / Hit@k- RAG Eval Pro same idea plus chunk-strategy switches, a toy rerank toggle, and a metrics table (md/csv) with short “how to read the table” notes

Not included / not claimed:

  • not a production RAG platform
  • not real-corpus benchmarks
  • no model weights, no API keys, no scraped course content
  • faithfulness remains N/A unless you wire a generator yourself

There is also a short discussion thread on r/LocalLLaMA if you want community critique on metric defaults:

https://www.reddit.com/r/LocalLLaMA/comments/1wg4gwm/tiny_stdlib_rag_retrieval_eval_harness_synthetic/

Soft CTA packs (optional)

If you want the ready-made kits instead of rebuilding from scratch:

Author: Dicardo9 / Moxuan Ding. Paid digital packs on Payhip; I made them. Educational use first — treat synthetic scores as practice, not as proof for production.

If you already have an eval set: what k and Hit@k vs Recall@k defaults do you actually use day to day? Critique welcome.

Top comments (1)

Collapse
 
jo-do profile image
Jo Do •

Starting with a deliberately dumb retriever is good teaching - when retrieval is the only moving part, precision@k actually means something. The honest scope note deserves a nod too: synthetic scores tell you whether your loop measures anything, not whether your production retrieval is good. The natural next step that keeps the spirit: swap one real embedding model in behind the same harness and watch the same three numbers move.