TL;DR: RAG metrics fall into three categories. Retrieval metrics (hit rate, recall@k, MRR, nDCG) tell you whether the system retrieves the right chunks. Generation metrics (faithfulness, answer relevancy, correctness) tell you whether the LLM used them well. Operational metrics (cost, latency, storage) tell you if you can afford to run it. This guide explains each metric with an example and tells you which ones to focus on for your setup.
There’s no shortage of RAG metrics. Libraries expose dozens of them, and dashboards show them all with the same font size. In practice, a handful do most of the work, and knowing which question each one answers matters more than tracking all of them.
Retrieval metrics
Retrieval metrics compare the chunks your retriever returns with the chunks that are known to contain the answer, listed in a golden dataset. Everything is computed on the top k results per question, then averaged over all questions.
To understand these metrics we will take a concrete example. Supposing that we have a golden dataset with the following question and expected chunks:
We configured our retriever to return the top 5 chunks for each question. The retriever returned these chunks for our question:
Let’s now calculate the metrics based on our example.
Hit rate@k: did we find anything useful ?
For this metric we set the value to 1 if at least one expected chunk is in the top k, else 0. Averaged over questions, it’s the share of questions where the LLM gets at least some of the information it needs.
Here we have #14 in the top 5, so hit rate = 1.
It’s the easiest metric to explain. But it ignores ranking and completeness.
Recall@k: how much of the answer did we find ?
Expected chunks retrieved divided by total expected chunks.
Here we have 1 of 2 expected chunks, so recall@5 = 1 / 2 = 0.5.
Recall@k matters when answers are spread across several chunks. Missing some chunks means the LLM works with half the facts, and that’s how you get incomplete answers.
Precision@k: how much of what we retrieved is useful?
Expected chunks retrieved divided by k.
In our example: 1 / 5, so precision@5 = 0.2.
Be careful with this metric. With a single expected chunk and k = 5, the best possible precision is 20%. It’s useful for comparing two setups, not as an absolute grade. It matters more when context is expensive or when irrelevant chunks visibly confuse your LLM.
MRR: how high is the first right chunk?
Mean Reciprocal Rank: 1 divided by the rank of the first expected chunk, 0 if none is found.
In our example: the first expected chunk is at rank 2, so MRR = 1/2 = 0.5.
This is the metric I look at first. LLMs pay more attention to the first passages they’re given, and many systems only use the top 1 to 3 chunks. Two models with the same hit rate aren’t equal if one puts the answer first and the other puts it fifth.
nDCG@k: how good is the whole ranking?
Normalized Discounted Cumulative Gain gives credit for every expected chunk found, with less credit the lower it’s ranked (the discount is logarithmic), and divides by the score of a perfect ranking. It ranges from 0 to 1.
In our example: one of two expected chunks at rank 2. Assuming binary relevance, DCG = 1/log₂(3) ≈ 0.63. The ideal ranking puts both at ranks 1 and 2: ideal DCG = 1 + 0.63 = 1.63. nDCG@5 ≈ 0.39.
nDCG@k try to answer both questions: “did we find all the chunks we needed” and “were they ranked high” in one number. It’s the standard metric for retrieval benchmarks (MTEB’s retrieval tasks report nDCG@10), and a good single number to report when answers often span several chunks.
Diagnostic retrieval metrics
Along with these core metrics, some patterns could be calculated to understand the retrieval performance better. These three patterns help you understand why a setup scores the way it does:
Hit rate by cut-off (hit rate@1, @2 … @k). A curve that starts low and climbs means the model finds the right area but doesn’t put the best chunk first: a reranker or a larger top k often helps.
Rank of the first relevant chunk, as a distribution. It separates near misses (rank 4 or 5 for top 5) from complete failures.
Score gap: the similarity of the best expected chunk minus the similarity of the best wrong chunk (the top distractor). A large
positive gap means stable rankings. A gap near zero means the right chunk and look alikes are almost tied, so small rewordings flip the result. Near-duplicate documents and boilerplate-heavy chunks are the usual causes.
Let’s now move to the next step: generation metrics.
Generation metrics
Once retrieval is solid, you evaluate the answer itself. These metrics are usually computed with an LLM as judge, using libraries such as Ragas:
Faithfulness (groundedness): are the answer’s claims supported by the retrieved context
Answer relevancy: does the answer address the question that was asked?
Answer correctness: does the answer match a reference answer?
Context precision and context recall (LLM-judged): retrieval-style metrics where a judge decides relevance instead of known chunk IDs.
A judge model has its own errors and biases, so scores can shift when you change the judge or its prompt. And a low faithfulness score doesn’t tell you whether the LLM ignored good context or never received it. Retrieval metrics on known chunk IDs answer that question.
Operational metrics
Quality is half the decision. For each retrieval setup, we could measure operational metrics such as:
Index cost: embedding the whole corpus once, and again each time you re-index or change models.
Cost per 1,000 queries: embedding user questions at your traffic.
Query latency: turning a question into a vector, plus the vector search (p50 and p95).
Vector storage: chunks × dimensions × 4 bytes for float32 vectors. A 3,072-dimension model needs twice the storage of a 1,536-dimension one.
Embedding is usually a small part of a RAG system’s cost compared with the LLM, but it matters at scale: large corpora, frequent re-indexing and high query volume.
The metrics in Truvec
Truvec focuses on the retrieval and operational layers. It grades embedding models and chunking strategies on your own documents, using a golden dataset you review and lock, so scores are deterministic and comparable over time. It doesn’t grade generated answers.
For every evaluation and every model in a comparison, Truvec reports:
Quality, at your top k (1 to 50, default 5)
- Hit rate
- Recall@k
- Precision@k
- MRR
- nDCG@k
- F1@k
Diagnostics
Hit rate by cut-off, from 1 to k, as a curve
Rank of the first relevant chunk, as a distribution, including misses
Score gap, mean relevant score and top distractor score
Cost, speed and storage
Index cost and cost per 1,000 queries, from the tokens actually billed and each model’s list price per million tokens
Billed by the current run
Query embedding latency, and vector search p50 / p95
Vector storage and dimensions
Per question
The Retrieval Inspector shows each question’s retrieved chunks in rank order, marked Ground truth or Distractor, with their similarity scores, plus the Expected but not retrieved chunks. Filter on Misses to see what each model misses. In a comparison, models are ranked by MRR, then recall, and the PDF report adds a head-to-head analysis and a matrix with the rank of the first relevant chunk for every question and every model.
Top comments (0)