Day 12/90: Before Improving Your RAG App, Establish a Baseline

Your RAG system retrieves the right documents. The answers look reasonable. The demo works.

But is the system actually improving?

Without a baseline, you can't reliably answer that question.

For Day 12 of my 90-day AI engineering journey, I'm exploring a fundamental principle of building reliable RAG systems:

Measure first. Optimize second.

1. Why a baseline matters

Imagine you're building a document Q&A system.

Your initial retriever uses dense vector search. You then introduce hybrid search, a reranker, and a different chunking strategy.

The answers seem better during manual testing.

But did retrieval quality actually improve? Did latency increase? Are you paying more per query for a marginal gain?

Without measuring the original system, you're comparing impressions—not performance.

2. Establish a baseline before changing anything

Start with a small, representative evaluation dataset.

For each question, record:

• The expected answer.
• The relevant source documents or chunks.
• The answer's supporting evidence.

Then evaluate your current system using metrics such as:

Retrieval quality

• Recall@K: Did the retriever find the relevant documents in its top K results?
• MRR: How highly ranked was the first relevant result?

Answer quality

• Faithfulness: Is the generated answer supported by the retrieved context?
• Answer relevance: Does the response actually address the question?

Operational performance

• Latency: How long does a query take?
• Cost per query: What does it cost to retrieve context and generate an answer?

Each metric answers a different question. A high retrieval score doesn't automatically mean the final answer is correct.

3. Run a controlled experiment

Suppose your initial evaluation produces these illustrative results:

Metric	Baseline
Recall@5	0.72
Faithfulness	0.80
Answer relevance	0.84
P95 latency	1.2 s

These numbers are hypothetical, not benchmark results.

Now you introduce a reranker and run the same evaluation dataset again.

If Recall@5 remains unchanged but faithfulness improves, the reranker may be helping the system prioritize more useful context—or the improvement may come from another part of the pipeline.

You need further analysis to identify the cause.

Also, don't rely on a single aggregate score. Inspect failed queries, compare retrieved chunks, and check whether improvements hold across different question types.

4. Change one thing at a time

A common mistake is changing chunk size, embedding models, retrieval strategy, prompts, and reranking simultaneously.

If performance improves, you won't know which change helped.

A more disciplined approach:

1. Establish a baseline.
2. Identify the dominant failure mode.
3. Change one component.
4. Run the same evaluation set.
5. Compare quality, latency, and cost.
6. Keep the change only if the trade-off is worthwhile.

For example, if relevant documents never appear in the top five results, rewriting the generation prompt probably won't solve the underlying retrieval problem.

Fix the failing stage before optimizing the next one.

5. The production takeaway

RAG evaluation isn't just about proving that a system works. It's about detecting regressions and making informed engineering decisions as the system evolves.

A new embedding model might improve retrieval while increasing latency. A reranker might improve answer quality while making every request more expensive.

The right choice depends on your application's requirements—not on which component sounds more advanced.

My takeaway:

If you can't measure the improvement, you can't confidently claim the optimization worked.

I'm curious how other AI engineers approach this:

When evaluating a RAG pipeline, which do you establish first—a retrieval baseline with Recall@K and MRR, or an end-to-end baseline covering answer quality, latency, and cost?