DEV Community: Abdesslem Amri

RAG evaluation metrics explained

Abdesslem Amri — Tue, 13 Oct 2026 19:51:57 +0000

TL;DR: RAG metrics fall into three categories. Retrieval metrics (hit rate, recall@k, MRR, nDCG) tell you whether the system retrieves the right chunks. Generation metrics (faithfulness, answer relevancy, correctness) tell you whether the LLM used them well. Operational metrics (cost, latency, storage) tell you if you can afford to run it. This guide explains each metric with an example and tells you which ones to focus on for your setup.

There’s no shortage of RAG metrics. Libraries expose dozens of them, and dashboards show them all with the same font size. In practice, a handful do most of the work, and knowing which question each one answers matters more than tracking all of them.

Retrieval metrics

Retrieval metrics compare the chunks your retriever returns with the chunks that are known to contain the answer, listed in a golden dataset. Everything is computed on the top k results per question, then averaged over all questions.

To understand these metrics we will take a concrete example. Supposing that we have a golden dataset with the following question and expected chunks:

We configured our retriever to return the top 5 chunks for each question. The retriever returned these chunks for our question:

Let’s now calculate the metrics based on our example.

Hit rate@k: did we find anything useful ?

For this metric we set the value to 1 if at least one expected chunk is in the top k, else 0. Averaged over questions, it’s the share of questions where the LLM gets at least some of the information it needs.

Here we have #14 in the top 5, so hit rate = 1.

It’s the easiest metric to explain. But it ignores ranking and completeness.

Recall@k: how much of the answer did we find ?

Expected chunks retrieved divided by total expected chunks.

Here we have 1 of 2 expected chunks, so recall@5 = 1 / 2 = 0.5.

Recall@k matters when answers are spread across several chunks. Missing some chunks means the LLM works with half the facts, and that’s how you get incomplete answers.

Precision@k: how much of what we retrieved is useful?
Expected chunks retrieved divided by k.

In our example: 1 / 5, so precision@5 = 0.2.

Be careful with this metric. With a single expected chunk and k = 5, the best possible precision is 20%. It’s useful for comparing two setups, not as an absolute grade. It matters more when context is expensive or when irrelevant chunks visibly confuse your LLM.

MRR: how high is the first right chunk?

Mean Reciprocal Rank: 1 divided by the rank of the first expected chunk, 0 if none is found.

In our example: the first expected chunk is at rank 2, so MRR = 1/2 = 0.5.

This is the metric I look at first. LLMs pay more attention to the first passages they’re given, and many systems only use the top 1 to 3 chunks. Two models with the same hit rate aren’t equal if one puts the answer first and the other puts it fifth.

nDCG@k: how good is the whole ranking?

Normalized Discounted Cumulative Gain gives credit for every expected chunk found, with less credit the lower it’s ranked (the discount is logarithmic), and divides by the score of a perfect ranking. It ranges from 0 to 1.

In our example: one of two expected chunks at rank 2. Assuming binary relevance, DCG = 1/log₂(3) ≈ 0.63. The ideal ranking puts both at ranks 1 and 2: ideal DCG = 1 + 0.63 = 1.63. nDCG@5 ≈ 0.39.

nDCG@k try to answer both questions: “did we find all the chunks we needed” and “were they ranked high” in one number. It’s the standard metric for retrieval benchmarks (MTEB’s retrieval tasks report nDCG@10), and a good single number to report when answers often span several chunks.

Diagnostic retrieval metrics

Along with these core metrics, some patterns could be calculated to understand the retrieval performance better. These three patterns help you understand why a setup scores the way it does:

Hit rate by cut-off (hit rate@1, @2 … @k). A curve that starts low and climbs means the model finds the right area but doesn’t put the best chunk first: a reranker or a larger top k often helps.

Rank of the first relevant chunk, as a distribution. It separates near misses (rank 4 or 5 for top 5) from complete failures.

Score gap: the similarity of the best expected chunk minus the similarity of the best wrong chunk (the top distractor). A large
positive gap means stable rankings. A gap near zero means the right chunk and look alikes are almost tied, so small rewordings flip the result. Near-duplicate documents and boilerplate-heavy chunks are the usual causes.

Let’s now move to the next step: generation metrics.

Generation metrics

Once retrieval is solid, you evaluate the answer itself. These metrics are usually computed with an LLM as judge, using libraries such as Ragas:

Faithfulness (groundedness): are the answer’s claims supported by the retrieved context
Answer relevancy: does the answer address the question that was asked?
Answer correctness: does the answer match a reference answer?
Context precision and context recall (LLM-judged): retrieval-style metrics where a judge decides relevance instead of known chunk IDs.

A judge model has its own errors and biases, so scores can shift when you change the judge or its prompt. And a low faithfulness score doesn’t tell you whether the LLM ignored good context or never received it. Retrieval metrics on known chunk IDs answer that question.

Operational metrics

Quality is half the decision. For each retrieval setup, we could measure operational metrics such as:

Index cost: embedding the whole corpus once, and again each time you re-index or change models.
Cost per 1,000 queries: embedding user questions at your traffic.
Query latency: turning a question into a vector, plus the vector search (p50 and p95).
Vector storage: chunks × dimensions × 4 bytes for float32 vectors. A 3,072-dimension model needs twice the storage of a 1,536-dimension one.

Embedding is usually a small part of a RAG system’s cost compared with the LLM, but it matters at scale: large corpora, frequent re-indexing and high query volume.

The metrics in Truvec

Truvec focuses on the retrieval and operational layers. It grades embedding models and chunking strategies on your own documents, using a golden dataset you review and lock, so scores are deterministic and comparable over time. It doesn’t grade generated answers.

For every evaluation and every model in a comparison, Truvec reports:

Quality, at your top k (1 to 50, default 5)

Hit rate
Recall@k
Precision@k
MRR
nDCG@k
F1@k

Diagnostics

Hit rate by cut-off, from 1 to k, as a curve
Rank of the first relevant chunk, as a distribution, including misses
Score gap, mean relevant score and top distractor score

Cost, speed and storage

Index cost and cost per 1,000 queries, from the tokens actually billed and each model’s list price per million tokens
Billed by the current run
Query embedding latency, and vector search p50 / p95
Vector storage and dimensions

Per question

The Retrieval Inspector shows each question’s retrieved chunks in rank order, marked Ground truth or Distractor, with their similarity scores, plus the Expected but not retrieved chunks. Filter on Misses to see what each model misses. In a comparison, models are ranked by MRR, then recall, and the PDF report adds a head-to-head analysis and a matrix with the rank of the first relevant chunk for every question and every model.

Ragas vs Truvec for RAG Retrieval Evaluation

Abdesslem Amri — Mon, 12 Oct 2026 12:30:00 +0000

I have built and debugged many RAG systems, and the most common failure I saw was not because of bad prompts or weak LLM models. Most failures were due to the right passage that was never retrieved or retrieved at the wrong rank. This post starts with a short guide on how retrieval evaluation works, then compares Ragas and Truvec and explains where each one fits.

Part 1: A short guide on RAG retrieval evaluation

A retrieval augmented generation (RAG) system answers questions in two steps, retrieval finds the passages (chunks) of your documents that are relevant to the question and generation gives those chunks to an LLM to write the answer.

The retrieval step has many moving parts such as:

Chunking: How documents are split into passages (size, overlap, by paragraph, by Markdown header, by function…).
The embedding model: Turns each chunk and each question into a vector.
The retrieval algorithm: How the closest chunks are found for example cosine similarity
Vector storage: How the vectors are stored (Qdrant, Weaviate, PGVector…).
The reranker: How the closest retrieved chunks are reordered
Top k: How many of the chunks to pass to the LLM.

If the chunk with the answer is at rank 9 and you send the top 5, the LLM will not be able to answer the question correctly. because the generation step only works with what retrieval hands it.

What a retrieval evaluation needs

To evaluate a retrieval process, you need three things. The documents or corpus, your system will search, chunked the way production chunks them. A golden dataset, test questions linked to the chunks that contain the answer (the ground truth) and the metrics to score where the expected chunks landed in each result.

The core retrieval metrics

To explain these metrics, let’s take a concrete example: Supposing you have a pdf about your company’s internal processes and you want to evaluate how you can find the relevant chunks for a given question. You already chunked the pdf into 50 chunks and stored them in a vector database and created a golden dataset with questions and their expected chunks.

Take a question whose answer lives in chunks #14 and #15. The retriever configured with top_k=5, returned in order these chunks #22, #14, #8, #41, #3. This means that the first expected chunk #14 is at rank 2. The table below calculates some of the core retrieval metrics for this example:

These are deterministic retrieval metrics, the same chunks, questions and model always give the same score.

Ground truth or LLM-as-judge (or JEV-as-judge)

There are two broad ways to decide whether retrieved chunks are relevant:

Reference-based or ground truth IDs: you know in advance which chunks answer each question. Scoring is exact and cheap, and you can compare models with confidence. The cost is building the golden dataset.
LLM-as-judge: an LLM reads the question, the retrieved chunks and the reference answer, and decides what’s relevant. It needs less setup and works on any pipeline, but every score depends on a judge model, its prompt and its own mistakes, and each run costs LLM calls.

Ragas leans on the second approach. Truvec uses the first one. That difference explains most of what follows.

Part 2: Evaluation with Ragas

Ragas is an open-source Python library for evaluating LLM applications, and it’s one of the most widely used tools for RAG evaluation. Using Ragas you can compute metrics such as:

Context precision and context recall: how relevant the retrieved contexts are, and how much of the reference they cover.

**Faithfulness: **whether the answer’s claims are supported by the retrieved contexts.

Response relevancy: whether the answer addresses the question.
Other metrics, such as noise sensitivity and context entity recall.

Most of these metrics use an LLM as judge. Recent versions also include non-LLM and ID based variants of context precision and recall when you supply reference contexts. Ragas can also generate synthetic test sets from your documents.

Where you can use Ragas:

You can use Ragas to evaluate a pipeline you already built, end to end, including rerankers, hybrid search and query rewriting. and calculate metrics like faithfulness, hallucinations, relevance.

What it asks of you:

With Ragas you should build and run the pipeline for every configuration you want to test. Comparing five embedding models means indexing your corpus five times, with five provider accounts and API keys. The LLM-judged evaluations depends on the judge model and cost tokens on each run.

Part 3: Evaluation with Truvec

Truvec is a hosted platform that evaluates the retrieval step on your own documents. You upload a sample of documents (Truvec chunks them based on your preferences) or chunks that you already have, and the platform builds a golden dataset for you, runs selected embedding models against it and scores where the expected chunks land.

Where you can use Truvec:

Side-by-side model comparison. Pick two or more models and run one comparison. Truvec supports multiple embedding models from OpenAI, Cohere, Voyage AI, Google, and open-source models (BGE, Nomic, mxbai, GTE, Qwen3 Embedding). You don't need any provider API key: Truvec runs the models on its own accounts and your plan includes the usage.

Calculate rank-based metrics: hit rate, recall, precision, MRR, nDCG and F1 at your top k, plus diagnostics: hit rate at every cut-off, rank of the first relevant chunk, and the score gap between the right chunk and the best distractor.

Calculate cost, speed and storage: index cost, cost per 1,000 queries, query embedding latency, vector search p50/p95 and vector storage for each model, computed from real token counts.

The Retrieval Inspector: For every question, see the ranked chunks, marked Ground truth or Distractor, and the Expected but not retrieved chunks.

A Playground to run any question against any supported model, with no API setup.
PDF reports with a recommendation, a leaderboard, a head-to-head analysis between models and a question-by-question matrix, for the decision record.

Truvec does not evaluate the answer your LLM writes, nor does it run your own pipeline.

Use Truvec when:

You’re choosing an embedding model for a new project and want evidence on your own documents, not a leaderboard rank.
You’re considering a migration to a new embedding model and need to know if it’s better enough to justify re-indexing everything.

Your answers are often wrong or vague and you suspect retrieval. The Misses filter usually tells you in ten minutes whether the problem is the model, the chunking, PDF extraction or the test questions.

You want to test a different chunking setup quickly without rebuilding your pipeline. You need a report a non-specialist can read.

Use Ragas when:

Retrieval is solid and you need to grade the generated answers: faithfulness, hallucinations, relevance.
You want to score the exact pipeline you run, including rerankers and query rewriting, inside your codebase.
Use both when you’re building a serious RAG product.

A workflow that works well:

Prototype retrieval in Truvec: pick the embedding model and chunking with a golden dataset and a comparison.
Build the pipeline with the winning setup.
Evaluate end to end with Ragas, focusing on the generation metrics.
When a new embedding model comes out or your corpus changes, go back to Truvec and re-run the comparison with your current model as the baseline.

Why isolating retrieval matters

When a RAG pipeline quality drops, you don’t know whether the retriever, the prompt or the LLM is to blame. Isolating retrieval metrics like hit rate@5 helps you identify the issue.

It also makes decisions cheaper. Comparing six embedding models with an LLM judge means six indexing pipelines and thousands of judge calls. Comparing them on a locked golden dataset is deterministic.

Originally published at https://truvec.dev on October 9, 2026.