Building ReportIQ: Evaluating RAG Retrieval Beyond Just Getting an Answer

When building a Retrieval-Augmented Generation (RAG) application, getting an answer from an LLM is only part of the problem.

A more important question is: Did the system retrieve the right information to answer the question?

I wanted to explore this problem hands-on, so I built ReportIQ, a document Q&A application focused on retrieval evaluation and LLM observability.

What is ReportIQ?

ReportIQ lets users interact with PDF documents through a RAG pipeline. Alongside document-based question answering, I worked on evaluating retrieval quality and inspecting the workflow behind the answers.

The goal is to look beyond the generated answer and better understand the retrieval process that supports it.

The Tech Stack

• Python: Core application and data processing
• Streamlit: User interface
• BGE-small-en-v1.5: Text embeddings
• FAISS: Vector similarity search
• LangSmith: LLM tracing and observability

Building the Document Pipeline

I processed 8 PDF documents into 721 chunks.

Chunking was an important part of the process because retrieval quality depends on how source information is divided and represented.

I used BGE embeddings to represent the chunks as vectors and FAISS to retrieve relevant passages for a user's query.

This gave me a foundation for experimenting with retrieval quality rather than relying only on whether a generated response sounded correct.

Evaluating Retrieval Quality

Rather than relying only on whether an answer looked reasonable, I evaluated retrieval against a set of 50 questions.

My latest recorded baseline achieved:

Metric	Result
Recall@1	66.7%
Recall@3	91.7%
Recall@5	91.7%
MRR@5	79.2%

These results come from a small evaluation set, so they should be treated as an initial baseline rather than proof of general performance.

The experiment gave me a more concrete way to investigate retrieval behavior and identify areas for improvement.

LLM Observability

Evaluating retrieval metrics is one part of understanding a RAG application. Inspecting the LLM workflow can provide additional context about what happens during execution.

I also worked with LangSmith to trace and inspect the LLM workflow.

What I Learned

Building ReportIQ helped me explore several practical aspects of RAG systems:

• Document processing and chunk quality
• Embedding models and vector search
• Retrieval evaluation using recall and reciprocal rank
• LLM workflow tracing and observability
• The importance of inspecting the pipeline rather than judging only the final answer

I'm continuing to improve ReportIQ and make the evaluation workflow more useful.

Try ReportIQ

• Live demo: https://reportiq.streamlit.app/
• Source code: https://github.com/Ronit-019/ReportIQ

I'd appreciate feedback from developers working on RAG applications.

Question for the community: What metrics or evaluation methods do you use to determine whether your RAG pipeline is retrieving the right evidence?

---

Built as a hands-on project to explore RAG retrieval, evaluation, and observability.