My RAG evaluation pipeline returned nan — here's what that taught me about Groq, RAGAS, and production LLM systems

Rajesh Chalise — Mon, 13 Jul 2026 18:54:38 +0000

I built four GenAI projects before this one: a PDF chatbot, a tool-calling exam-prep agent, a manual ReAct agent built from scratch with LangGraph, and a multi-agent research assistant. All four worked. But "it works" was never something I could actually prove — I just read the output and decided it looked reasonable.

Project 5 was about fixing that. I built a standalone evaluation pipeline using RAGAS to score my RAG system on four metrics: Faithfulness, Answer Relevancy, Context Precision, and Context Recall. Getting it running taught me more about production LLM systems than any of the previous four projects — mostly because of what went wrong along the way.

What I built

A pipeline that: fetches ArXiv paper abstracts on a topic, chunks them, uses an LLM (Groq's llama-3.3-70b-versatile) to auto-generate question/ground-truth-answer pairs from those chunks, builds a FAISS index over the same chunks using sentence-transformers/all-MiniLM-L6-v2, runs each generated question through a simple retrieve-then-answer RAG pipeline, and finally scores the results with RAGAS.

Tech stack: Python, LangChain, FAISS, HuggingFace embeddings, Groq, the arxiv Python library, RAGAS.

Decision 1: why I built this separate from my multi-agent project

My Project 4 was a multi-agent supervisor system — dynamic FAISS indexing, agents routing between each other, shared state. If I'd bolted RAGAS evaluation onto that, a bad score wouldn't tell me anything useful, because I'd be evaluating the entire agent orchestration at once — routing decisions, agent handoffs, retrieval, and generation, all tangled together. RAGAS needs a controlled, static setup: fixed corpus, fixed Q&A pairs, fixed retriever config, so that changing one variable (chunk size, embedding model, top-k) and re-running actually tells you something. A standalone pipeline was the only way to get that isolation.

Decision 2: why ground-truth Q&A pairs come from chunks, not full abstracts

RAGAS needs reference answers to compute two of its four metrics — Context Precision and Context Recall both require a ground truth to check retrieval against. My first instinct was to generate these Q&A pairs from full abstracts. That's wrong for a subtle reason: your evaluation should mirror your production setup. At query time, my retriever pulls back chunks, not full abstracts. If my ground-truth answer was generated from the whole abstract, it might reference information that got scattered across two or three separate chunks once indexed — and then Context Recall would penalize my retriever for not doing something structurally difficult, not for being genuinely bad.

So I generate one question and one ground-truth answer per chunk, using the exact same chunking logic I use to build the FAISS index. Same granularity everywhere, no artificial mismatch.

The debugging story: from nan to a rate limit I didn't see coming

Here's where it got interesting.

First run — real numbers came back:

faithfulness: 1.0000
answer_relevancy: 0.7936
context_precision: nan
context_recall: 0.8571

Faithfulness and recall looked great. But context_precision: nan isn't a real score — it means the metric failed to compute, not that precision was undefined by design. My first guess was a bug in my dataset formatting. It wasn't.

Scrolling back through my terminal, I found the real cause:

Exception raised in Job[8]: TimeoutError()
Exception raised in Job[18]: TimeoutError()
LLM returned 1 generations instead of requested 3. Proceeding with 1 generations.

RAGAS fires off many LLM calls in parallel to speed up evaluation — fine on a provider like OpenAI, but Groq's free tier has tighter concurrency limits, and several calls were timing out before they ever got a response. When enough sub-calls for a metric fail, RAGAS can't compute a valid average, so it returns nan instead of a wrong number.

Fix attempt one: I added RunConfig(max_workers=1, timeout=180) to force RAGAS to run everything sequentially instead of in parallel. This fixed the timeout issue, but exposed a second, different problem — Answer Relevancy started throwing:

BadRequestError(Error code: 400 - {'error': {'message': "'n' : number must be at most 1", ...}})

Answer Relevancy's internal mechanism generates three reverse-engineered questions per answer in a single call (n=3) and compares their embedding similarity to the original question. Groq's API has a hard rule: n can only ever be 1. No amount of concurrency tuning fixes this — it's a structural incompatibility. The real fix was reconfiguring the metric itself:

pythonfrom ragas.metrics import AnswerRelevancy
answer_relevancy_metric = AnswerRelevancy(strictness=1)

This forces the metric to request exactly one generation per call instead of three, at the cost of slightly less statistical robustness per row.

Then, a third problem — this one wasn't a bug at all. Partway through a full 12-row evaluation:

RateLimitError(Error code: 429 - {'error': {'message': 'Rate limit reached for model
llama-3.3-70b-versatile ... on tokens per day (TPD): Limit 100000, Used 99617,
Requested 535 ...'}})

I'd simply exhausted Groq's free-tier daily token quota — not from this one evaluation run alone, but from the cumulative cost of every debugging run, test call, and Q&A generation pass I'd made that day. RAGAS makes dozens of LLM calls per row across four metrics; multiply that by testing iterations, and a 100k-token daily budget disappears fast. No config fixes a hard account limit — the only real fix is patience (wait for the daily reset) and being more deliberate about testing on small slices before running full evaluations.

What I'd tell someone starting this

Three lessons, in order of how expensive they were to learn:

nan in an evaluation metric is a signal to check your logs, not your dataset. My first instinct was to re-check my data formatting. The real cause was infrastructure — timeouts — not logic.
Free-tier LLM APIs have constraints RAGAS wasn't built around. n=3 requests, high concurrency, generous rate limits — RAGAS assumes a provider like OpenAI can absorb all of this. Cheaper/free providers often can't, and the fixes live in RAGAS's RunConfig and per-metric configuration, not in your own pipeline code.
Token budgets are a real constraint, not just a cost line item. When you're iterating on a pipeline that makes 100+ calls per full run, test on 2-3 rows while debugging, and save your full run for when you're confident the logic is correct.

Results

After working through the timeout, the n=3 incompatibility, and finally a daily token quota limit, here's what the pipeline reported on 12 auto-generated questions over 5 ArXiv abstracts on attention mechanisms:

Faithfulness: 1.0000
Answer Relevancy: 0.7345
Context Precision: 0.7500
Context Recall: 0.8000

Faithfulness — 1.0. Every claim in every generated answer traced back to the retrieved context. Zero hallucination across all 12 questions. This is exactly what I'd hope for given the prompt explicitly instructs the model to say "the answer is not present in the text" rather than invent one — and I watched it do exactly that in practice. For one question about "DiNAT variants compared to strong baselines," the retriever pulled the wrong chunks, and instead of guessing, the model correctly said it couldn't find the answer. That's a faithful failure, not a hallucinated success — and it's precisely the distinction this metric is built to catch.

Context Recall — 0.80. Roughly 1 in 5 pieces of information that should have been retrievable, weren't. This lines up with what I saw directly in my raw output: the ground-truth chunk for "DiNAT variants" existed in my FAISS index (it showed up correctly for a different question), but never got surfaced for the question it actually belonged to. That's a retrieval problem, not a generation problem — no amount of prompt engineering fixes it. The fix lives upstream, in retrieval tuning (top-k, embedding model, or reranking).

Context Precision — 0.75. About a quarter of retrieved chunks, across all questions, weren't actually relevant to the specific question asked — even when they were topically "about attention mechanisms" in a loose sense. This is the noise problem: my retriever sometimes pulls in things that are semantically nearby but not answer-bearing.

Answer Relevancy — 0.73. The softest score of the four, and I'd read it with a grain of salt — this metric normally averages 3 reverse-engineered questions per answer, but I was forced to run it at strictness=1 (a single reverse-engineered question, no averaging) because Groq's API rejects any request asking for more than one completion at a time. A single-sample measurement is noisier than a 3-sample average by design, so this number likely understates true relevancy somewhat.

The overall picture: my generation step is trustworthy — it doesn't make things up, and it's honest when it can't answer. My retrieval step is the weaker link — it's missing roughly a fifth of what it should find, and pulling in some noise along the way. That's a genuinely useful, actionable diagnosis, and it's exactly the kind of insight "I read the output and it looked fine" could never have given me.

What's next

Two concrete next experiments, both aimed at the 0.80 recall / 0.75 precision gap rather than the generation side, since generation is already clean:

Increase top_k and re-run the same 12 questions to see if Context Recall improves — a direct test of whether the missing chunks simply weren't making the cutoff, versus not being found at all.
Try a panel of judges instead of a single LLM judge for Answer Relevancy, now that I understand LLM-as-a-judge is itself an active area of research with known consistency limitations — averaging across a couple of different models might stabilize a score that's currently based on a single sample due to Groq's n=1 constraint.

Both are measurable, both are cheap to try, and both are exactly the kind of question a proper evaluation pipeline is supposed to let you answer instead of guess at.

DEV Community: Rajesh Chalise

My RAG evaluation pipeline returned nan — here's what that taught me about Groq, RAGAS, and production LLM systems