I. Introduction: What Agent and RAG Evaluation Is
Have you ever built a system that worked flawlessly in testing but fell apart the moment it hit production?
Maybe you shipped an AI agent that answered everything correctly in your sandbox and even passed your evaluations, yet once users got hold of it, it started failing in subtle, expensive ways.
Or you wired a RAG component into a multi-agent system, watched it work perfectly in your test harness, and then realised in production that the RAG wasn't being referenced or used correctly at all.
The real cost of an unreliable agent
No matter how costly it is for LLMs to be unreliable in production, it can cost you:
- Reputation
- Your business
- Time
- Your valuable customers
- And it can even lead to legal action
People still prefer to vibe-check, make weak evaluations, or follow a happy path when running their evaluations. This can cause an LLM to work very well in a test environment with predictable answers but fail in the real world. Here are some examples of when that failure happened.
II. When AI Failures Get Expensive: Real-World Case Studies
One wrong answer from a customer service bot was enough to end up in court.
Not even a working AI agent (or a "reliable" agent, an LLM, or a RAG) is safe from being costly in a production environment. Let me tell you about the Moffatt v. Air Canada case.
Air Canada had a system built on top of an LLM, mainly for customer service. After his grandmother died, Mr. Moffatt used Air Canada's chatbot to ask about bereavement fares. The chatbot told him he could book his flight at full price and apply for the discounted bereavement rate retroactively within 90 days. He did exactly that, and Air Canada refused the refund, pointing out that its actual policy didn't allow retroactive bereavement claims. Moffatt took Air Canada to the BC Civil Resolution Tribunal and won.
Another scenario: a lawyer wanted to use ChatGPT as a reference when putting together a court case. The references and case law it gave him were inaccurate and filled with hallucinations. It ended up costing that lawyer a fine of over $5,000. There are many cases like this. You can look up the link I've shared on some of the legal cases around hallucinations, and take a look at this graph:
Refer to https://www.damiencharlotin.com/hallucinations/?graphs=1
The number of hallucination-related legal cases keeps climbing every year.
III. The Promise of the Article
So what can I promise with this article?
You'll understand what causes LLM agents to hallucinate and why that can cost you real money.
You'll learn, step by step, how to build a more reliable AI agent and run meaningful evaluations, whether you're at a startup or a large engineering org.
You'll get practical tools and resources you can use to go deeper into LLM and agent evaluation.
Let's get started.
Why agent and RAG evaluation is harder than testing a single LLM
Testing a single LLM is relatively straightforward. Often, you can improve prompts or compare outputs against a known "gold" answer. That's manageable because an LLM is essentially a next-token predictor.
Testing multi-agent systems or RAG pipelines is much harder. They have multiple steps, dynamic behaviour, and sometimes autonomous decision-making. You don't always know the exact output you want, or even what the "correct" path to that output should be. There can be many valid paths through the system, which makes it much harder to design robust evaluations.
Still, we have to try because the consequences of shipping untested or poorly evaluated AI systems are now much higher.
Common Types of Failures
a. Agentic logic and execution failures
Three types worth calling out here:
Infinite action loop. Quite common. This is where an agent keeps calling a tool but keeps getting an error and keeps repeating that same failed call over and over, eating your token budget.
Type mismatch. The tool's parameters aren't being sent as expected. Maybe an email tool is expecting a string but keeps getting an integer, so the application hard-crashes or keeps failing. Imagine that happening in production.
State desynchronization and memory loss. In a complex multi-agent workflow, the agent drops critical context required to answer the prompt properly and ends up giving the user a vague, generic answer that ignores the full original prompt.
b. RAG & retrieval failures
The most common one is garbage in, garbage out. You've given it poor information, done a poor chunking strategy, or fed it poor data. You end up retrieving irrelevant documents, and sometimes it will say it doesn't know. Worse, it will confidently give you an answer that has nothing to do with what you expected.
An example: it gives you an answer for, say, a funeral policy FAQ, but the wrong answer (a hallucination), because even though you ingested the FAQs, they weren't chunked and vetted properly into the system.
Another common one is the "lost-in-the-middle" effect. This happens when you've provided a very large context, bloating the prompt. Due to that massive context window, the model will simply ignore some of the user's actual instructions and focus on either the very beginning or the very end of the prompt. Obviously, this results in an answer that doesn't quite make sense.
c. Silent failures
These are the worst-case-scenario type of failures, because sometimes you won't even notice them, especially when you're doing evaluations or vibe checks. They only become visible once, say, a power user runs into one, even though the system has already been in production for a while. Unlike traditional software, it won't throw a loud break. You'll probably get a status 200, and everything will look flawless.
The agent quietly executes every update, but maybe it's updating the wrong client record in the DB. Another scenario: it's supposed to send an email to Person A, but instead sends it to Person B, an unrelated prospect. These are the silent failures that are hardest to catch, because the system still "works." It's still sending; it's still updating. It's just doing it on the wrong data.
IV. Deep Dive: What Causes AI Agents to Fail
In practice, most failures come from a few recurring patterns:
- Not testing at all
- Testing only the happy path
- Passing evals but still failing in production
- Not having a continuous eval pipeline
1. Not testing at all
This doesn't necessarily mean you don't test at all. Most AI engineers (at least 90% of the engineers I've dealt with) put most of their effort into making sure everything works, and only a small fraction of their time, maybe 10% or less, goes toward testing.
What they usually do instead is a vibe check: throw in maybe 10-20 prompts and just see if the system produces an output. Sometimes they don't even check the quality of that output:
- Is it fair?
- Is it legally risky?
- Is it following the prompt?
- Is anyone scoring that response?
2. Testing only the happy path
The happy path is where tests actually happen, but they're only happy-path tests. There are no edge cases, no red teaming done, and the questions are not adversarial. Everything is quite easy. Things like "What's my name?" or "What is X?" Predictable stuff.
The happy path will always pass, and it's only when the edge cases show up in production that people panic.
3. Passing evals but still failing in prod
This is the scary one, because this is where you've put in the work and done proper evaluations in the past. So why is it still failing? This is where most AI engineers get caught out. A few reasons:
We might not be looking at the correct thing. We might just be looking at the final output. If it's an AI agent, you also want to look at whether it followed the right steps to get to that answer. "Correct" and "acceptable" might be different things.
This is what I call the wrong-path-to-right-answer problem: you got the right answer, but through the wrong process. For example, in a weather application, instead of using the weather API, the agent might hallucinate the answer or use a completely different tool to "find" the weather in San Francisco. Without cross-checking, you'd assume the agent is working correctly.
This is shortcut behavior. Without proper reasoning and the correct steps, it can still produce a correct answer. The problem is that it won't always produce the right answer because it isn't consistently using the tool you actually wanted it to use.
Same right answer, but only one of these two paths actually called the weather API.
The agent skips a required step. There might be steps the agent is required to follow. For example, before sending an email to a prospect or lead, you might want it to:
- Find details about that prospect.
- Call the function that pulls that prospect's profile.
- Use a prompt that drafts and sends the email.
- After sending, update the pipeline so the prospect moves from "lead" to "outbound lead."
The problem is that the agent might send the email and update everything else, but skip the step of actually finding the prospect's details first, so it's working off incomplete context. That can cost your business a lot if you're in the business of securing new prospects.
The high-confidence wrong answer. This is where the agent is very confident, but totally wrong. This ties back to RAG: if your data wasn't chunked properly, the retriever can still pull documents, and the agent can still treat them as relevant to the prompt, even when they're not.
Everything appears to work, but the answer is wrong. An example: a large organization with FAQ documents for different products, say motor insurance and microloans, where only the microloan documents were chunked and indexed properly. A customer service agent works across all of those entities at once could confidently give a wrong answer to someone asking about their motor insurance.
Next, I will walk you through how to design a eval workflow that catches these failure modes before they hit production.
V. The Ideal Workflow for Dealing with Agents and RAG
This is the workflow you'd use if you want to stop vibe-checking and actually turn your AI project from a hobby into a solution that has real impact in production and is highly reliable.
Six steps, one loop: evaluation is a flywheel, not a one-time checklist.
1. Build a custom evaluation dataset
It might be tempting to purchase an existing dataset or use an open-source one. The issue is these datasets aren't fully aligned with what you're building in most cases; they have different benchmarks. It's better to curate your own dataset from your own real inputs, or generate something synthetic based on your own knowledge base.
2. Turn traces into a dataset
If your application is already in production, for example a complex agent you're already running traces on using DeepEval, Langfuse, or LangSmith, you can use those traces to build a dataset that's more aligned with reality: the edge cases, what's been catastrophic, and what's succeeded.
3. Pick the right balance of eval metrics
Don't start off with more than 10 metrics in most cases. Pick just a few, because you're not trying to boil the ocean. Things worth considering:
- Layer 1: deterministic metrics / code-level checks. Did it run? Did it return valid JSON? Was the latency low?
- RAG-specific metrics. Did it retrieve the proper context, and was the answer faithful to it?
- Layer 3: LLM as a judge. Using a separate, isolated model to score subjective quality against a strict rubric.
4. Run the evals
Now for the fun part: executing your evaluations systematically. Use different evaluation and orchestration tools (DeepEval, Ragas, LangSmith, etc.), depending on what you prefer and know, to run your target agent against the dataset you prepared.
5. Add evals to your pipeline (CI/CD)
Offline evaluations are great, but you also want online evaluations and a pipeline that treats your prompts and agent workflows like any other source code. When you change your model, say from GPT-5.5 to GPT-5.6, you want to be able to test things like faithfulness and see if the score dropped or improved, automatically, without human involvement, triggered on every PR.
6. Tracing feeding back into evals
This is what I call the flywheel. You want your production loop to be complete, able to update your dataset based on what's happening in real time, based on your traces. Rule number one: never let your evaluation dataset stay static.
Common LLM Eval Mistakes
Here are some common LLM evaluation mistakes teams make, especially early on. I call this the vibe-check trap.
Four traps that quietly sink most eval efforts before they even start.
Doing too much, too early. It's tempting to over-engineer the early stages of evaluation, 20 metrics, 100,000 synthetic data points, but that just burns your tokens and your team's engineering time. It's better to start small: 20-25 highly curated test cases with 3-4 high-value metrics tied to your actual business logic.
Focusing on the happy path (and ignoring operational realities). It's easy to write perfectly formatted questions with easy answers and test against those. We tend to forget that production gets messy, and end up testing the easy part while ignoring the adversarial edge cases. The agent looks great on paper, but the test isn't actually testing what will happen in production.
Not having a workflow, and relying on "vibe checks." This is the biggest and most common AI engineering trap: a handful of prompts tested by yourself in a playground, "yeah, looks good," then straight to production. No edge cases, no red teaming, just a subjective vibe check. It's unscalable, and it more or less guarantees silent failures down the line.
Having fixed evals that don't evolve with production. Production isn't just messy, it's dynamic and constantly changing. Models change, models drift, APIs change, user behavior changes. Testing the same 25-item dataset for a year after it's gone stale won't give you a sustainable long-term approach. What you need is a production-grade system that monitors change (those traces we talked about earlier) and feeds them back into the dataset.
VI. Different Tools You Can Use for Agent Evaluation
Three jobs, three tool categories: build it, score it, watch it.
Agent orchestration with built-in state: LangGraph (excellent for testing specific nodes), AutoGen, CrewAI.
RAG & agent metrics: Ragas, DeepEval, TruLens, LangSmith.
Agent tracing & observability: LangSmith (crucial for viewing multi-step agent thought processes), Datadog LLM Observability, DeepEval.
VII. Resources
LangSmith Evaluation - Docs by LangChain
docs.langchain.com
LangSmith evaluation allows for both offline and online evaluations. It's a good resource and tool if you're a fan of the LangChain ecosystem, like me.
Introduction | DeepEval - The LLM Evaluation Framework
DeepEval is the open-source LLM evaluation framework for testing and benchmarking LLM applications.
deepeval.com
DeepEval also provides great resources on the topic and tooling around LLM evaluation. I find it to be a very competent resource that will change how you see and do evaluations.
Further reading:
- Chapter 3, Evaluation Methodology, in AI Engineering by Chip Huyen.
- Chapter 10, Evaluating LLM Applications, in Prompt Engineering for LLMs by John Berryman & Albert Ziegler.
- Chapter 7, Evaluating LLMs, in LLM Engineer's Handbook by Paul Iusztin & Maxime Labonne, for a practical, hands-on take.
References
- Moffatt v. Air Canada, 2024 BCCRT 149 (CanLII). British Columbia Civil Resolution Tribunal. https://canlii.ca/t/k2spq
- McCarthy Tétrault, "Moffatt v. Air Canada: A Misrepresentation by an AI Chatbot." https://www.mccarthy.ca/en/insights/blogs/techlex/moffatt-v-air-canada-misrepresentation-ai-chatbot
- American Bar Association, "BC Tribunal Confirms Companies Remain Liable for Information Provided by AI Chatbot," Business Law Today, February 2024. https://www.americanbar.org/groups/business_law/resources/business-law-today/2024-february/bc-tribunal-confirms-companies-remain-liable-information-provided-ai-chatbot/
- Mata v. Avianca, Inc., 678 F. Supp. 3d 443 (S.D.N.Y. 2023). https://en.wikipedia.org/wiki/Mata_v._Avianca,_Inc.
- Legal Clarity, "Mata v. Avianca: Fake Cases, ChatGPT, and Sanctions." https://legalclarity.org/what-happened-in-the-mata-v-avianca-case/
- Damien Charlotin, AI Hallucination Cases Database (tracking legal cases involving generative AI hallucinations). https://www.damiencharlotin.com/hallucinations/?graphs=1
- LangChain, LangSmith Evaluation Documentation. https://docs.langchain.com/langsmith/evaluation
- DeepEval, Tutorial Introduction. https://deepeval.com/tutorials/tutorial-introduction
- Huyen, Chip. AI Engineering, Chapter 3: Evaluation Methodology. O'Reilly Media.
- Berryman, John & Ziegler, Albert. Prompt Engineering for LLMs, Chapter 10: Evaluating LLM Applications. O'Reilly Media.
- Iusztin, Paul & Labonne, Maxime. LLM Engineer's Handbook, Chapter 7: Evaluating LLMs. Packt Publishing.
Top comments (0)