Abdesslem Amri

Posted on Nov 14

RAG retrieval benchmarking and optimization

#ai #rag #truvec #programming

Your RAG is only as good as its retrieval. Here's how to benchmark it.

A retrieval miss doesn't look like an error. When the right passage isn't retrieved, the LLM still answers from the wrong data. Nothing in your logs points at retrieval.

In this article I will show you how to use Truvec to debug and optimize your retrieval.

The 5 knobs

Everything before the LLM changes which chunks come back:

Chunking
Embedding model
Dense or hybrid search (BM25 + vectors)
Reranker
Top k

Your contracts, support articles and code have their own vocabulary. You can not rely on other people setup to build your own retrieval system.

1. Get a golden dataset

Questions about your docs, each tied to the chunk that answers it.

UI: Data Upload → Golden Set → Create Dataset → Generate Dataset. Review the questions, fix the bad ones, then Commit Dataset. Committed means locked: every run uses the same questions.

API:

API=https://app.truvec.dev
AUTH="Authorization: Bearer $TOKEN"   # from POST /api/auth/login

curl -s -X POST "$API/api/v1/documents/upload?project_id=$PROJECT" -H "$AUTH" \
  -F "file=@help-center.pdf" -F "chunking_strategy=recursive" \
  -F "chunk_size=1000" -F "chunk_overlap=200"

DATASET=$(curl -s -X POST $API/api/v1/golden-datasets -H "$AUTH" \
  -H "Content-Type: application/json" \
  -d "{\"project_id\": $PROJECT, \"name\": \"v1\", \"number_of_questions\": 20}" | jq .id)

curl -s -X POST $API/api/v1/golden-datasets/$DATASET/commit -H "$AUTH"

2. Benchmark the models

UI: Comparison → Compare models → pick the dataset, tick the models, Run Comparison.

Look at that table. Same hit rate: 90% for both. MRR splits them: 0.900 vs 0.850. And the winner is the one priced at $0.020 per 1M tokens, not $0.100.

Hit rate says the right chunk is in the top k. MRR says how high. If you pass 3 chunks to the LLM, that's the one to watch.

API:

curl -s -X POST $API/api/v1/benchmarks -H "$AUTH" \
  -H "Content-Type: application/json" \
  -d "{\"project_id\": $PROJECT, \"golden_dataset_id\": $DATASET,
       \"name\": \"small vs large\", \"model_ids\": [1, 2], \"top_k\": 5}"

3. Optimize: one change at a time

Keep one baseline. Add hybrid search in one configuration, a reranker in another. Same questions.

UI: Comparison → Compare strategies. The Hybrid search and Reranker buttons copy the baseline and change one thing.

API: the first variant is the baseline.

curl -s -X POST $API/api/v1/benchmarks -H "$AUTH" \
  -H "Content-Type: application/json" -d "{
  \"project_id\": $PROJECT, \"golden_dataset_id\": $DATASET,
  \"name\": \"Does a reranker help?\", \"top_k\": 5,
  \"variants\": [
    {\"model_id\": 1},
    {\"model_id\": 1, \"pipeline\": {\"retrieval\": \"hybrid\"}},
    {\"model_id\": 1, \"pipeline\": {\"reranker\": {\"provider\": \"cohere\", \"model\": \"rerank-v3.5\"}}}
  ]
}"

Then read each one against the baseline:

curl -s $API/api/v1/benchmarks/$BENCHMARK -H "$AUTH" \
  | jq '.results.comparison[] | {config: .label, mrr,
        mrr_delta: .vs_baseline.mrr_delta,
        fixed: .vs_baseline.fixed, broken: .vs_baseline.broken}'

fixed: questions the baseline missed and this setup found.
broken: the opposite.

Fixing 4 and breaking 3 is not the same as fixing 1 and breaking none, even if the averages match.

What to expect

Hybrid shines on product names, error codes and SKUs. On plain prose it can add nothing, or hurt.
A reranker can make a cheap model beat an expensive one. It can also cost more per query than the embedding.
Read the questions a change broke before you ship it.

Put it in CI

It's all API calls, so run it when your documents change or before you switch models. Retrieval first, generation after.

Free plan, no card, no provider keys to set up. About 15 minutes to a first leaderboard on your own docs.

👉 https://truvec.dev
API docs: https://docs.truvec.dev

DEV Community