How an execution tree helped me find a one-line field-mapping bug in a customer-support RAG pipeline
The Problem: Retrieval “Worked,” But the Agent Still Said “I Don’t Know”
I was building a customer-support agent for a SaaS product. The setup was familiar:
- Next.js API route for chat
- Vercel AI SDK for streaming responses
- Pinecone for vector search over product docs
- GPT-4o for answer generation
In testing, it looked fine. In production-like runs, it failed in the most frustrating way possible.
User: “How do I reset my password?”
Agent: “I don’t have information about that. Please contact support.”
The docs had a page called password-reset.md. It was not vague. It explained the exact flow: click “Forgot password,” enter your account email, check your inbox, follow the reset link.
So I added logs.
console.log("Retrieved chunks:", chunks.length);
console.log("Similarity scores:", chunks.map((chunk) => chunk.score));
console.log("Context length:", context.length);
The logs were technically useful:
Retrieved chunks: 5
Similarity scores: [0.89, 0.85, 0.81, 0.77, 0.73]
Context length: 0
But they did not answer the real question.
Retrieval seemed to work. Scores were high. The context was empty. Somewhere between “found the right documents” and “called the model,” the useful information disappeared.
I could keep adding console.log statements, or I could inspect the pipeline as a pipeline.
That is where AgentInspect helped.
The Setup: A Simple RAG Route With Explicit Steps
Here is a simplified version of the route.
The important part is not the exact Pinecone or model setup. The important part is that each stage of the RAG pipeline becomes a named step:
- generate an embedding
- retrieve documents
- summarize retrieval safely
- build context
- summarize context safely
- generate the answer
I do not persist raw retrieved document text into the trace. That is intentional. The trace records metadata that is useful for debugging without dumping prompts, private user text, or internal docs into the artifact.
// app/api/chat/route.ts
import { embed, streamText } from "ai";
import { openai } from "@ai-sdk/openai";
import { inspectRun, step } from "agent-inspect";
// Assume this is your initialized vector index.
import { docsIndex } from "@/lib/docs-index";
type ChatMessage = {
role: "user" | "assistant" | "system";
content: string;
};
type RetrievedChunk = {
content: string;
score: number;
source: string;
};
async function note(
name: string,
metadata: Record<string, unknown>,
): Promise<void> {
await step(
name,
async () => undefined,
{
type: "state",
metadata,
},
);
}
export async function POST(req: Request) {
const { messages }: { messages: ChatMessage[] } = await req.json();
const userMessage = String(messages.at(-1)?.content ?? "");
return inspectRun(
"support-agent",
async () => {
const { embedding } = await step(
"generate-query-embedding",
() =>
embed({
model: "openai/text-embedding-3-small",
value: userMessage,
}),
{
metadata: {
model: "openai/text-embedding-3-small",
},
},
);
await note("embedding-summary", {
dimensions: embedding.length,
});
const chunks = await step(
"retrieve-docs",
async (): Promise<RetrievedChunk[]> => {
const queryResponse = await docsIndex.query({
vector: embedding,
topK: 5,
includeMetadata: true,
});
return queryResponse.matches.map((match) => ({
// This was the bug:
// our index stored the document body in `metadata.content`,
// but this code read `metadata.text`.
content: String(match.metadata?.text ?? ""),
score: Number(match.score ?? 0),
source: String(match.metadata?.source ?? "unknown"),
}));
},
{
type: "tool",
metadata: {
toolName: "pinecone.query",
topK: 5,
},
},
);
await note("retrieval-summary", {
matches: chunks.length,
topScore: chunks[0]?.score,
emptyContent: chunks.filter((chunk) => chunk.content.length === 0)
.length,
sources: chunks.map((chunk) => chunk.source),
});
const context = await step("build-context", async () => {
return chunks
.filter((chunk) => chunk.score > 0.7)
.map((chunk) => chunk.content)
.filter(Boolean)
.join("\n\n");
});
await note("context-summary", {
contextChars: context.length,
contextChunks: chunks.filter(
(chunk) => chunk.score > 0.7 && chunk.content.length > 0,
).length,
});
return step.llm("gpt-4o", async () => {
const result = streamText({
model: openai("gpt-4o"),
system: [
"You are a helpful support agent.",
"Answer from the provided documentation context when it is available.",
"",
"Documentation context:",
context || "(no retrieved context)",
].join("\n"),
messages,
});
return result.toUIMessageStreamResponse();
});
},
{
traceDir: "./.agent-inspect",
redactionProfile: "local",
},
);
}
This is still ordinary application code. AgentInspect is not replacing the RAG pipeline. It is giving each boundary a name and writing a local trace under .agent-inspect/.
The Breakthrough: The Tree Showed Where the Data Disappeared
After running the same failing query, I inspected the local trace.
npx agent-inspect list --dir ./.agent-inspect
npx agent-inspect view <run-id> --dir ./.agent-inspect --verbose
npx agent-inspect report <run-id> --dir ./.agent-inspect
The tree made the problem obvious.
support-agent (1.2s) ✓
├─ generate-query-embedding (312ms) ✓
├─ embedding-summary (1ms) ✓
│ metadata:
│ dimensions: 1536
├─ tool:retrieve-docs (428ms) ✓
│ metadata:
│ toolName: pinecone.query
│ topK: 5
├─ retrieval-summary (1ms) ✓
│ metadata:
│ matches: 5
│ topScore: 0.89
│ emptyContent: 5
│ sources:
│ - password-reset.md
│ - account-login.md
│ - user-settings.md
│ - security-faq.md
│ - email-delivery.md
├─ build-context (1ms) ✓
├─ context-summary (1ms) ✓
│ metadata:
│ contextChars: 0
│ contextChunks: 0
└─ llm:gpt-4o (530ms) ✓
That was the bug.
Retrieval was not failing. Pinecone returned five strong matches, and the top source was exactly the page I expected: password-reset.md.
But every retrieved match had empty content.
That meant the failure happened in the mapping layer:
content: String(match.metadata?.text ?? "")
The index did not store the body in metadata.text. It stored it in metadata.content.
Every step had “succeeded,” so there was no exception to catch. The embedding call succeeded. The vector search succeeded. The context builder succeeded. The model call succeeded.
The pipeline produced the wrong answer because valid-looking intermediate data was empty.
That is the kind of bug flat logs make painful. You can eventually find it, but you usually find it by scattering logs across every boundary. The execution tree showed the causal path in one place.
The Fix
The fix was one field name.
return queryResponse.matches.map((match) => ({
content: String(match.metadata?.content ?? ""),
score: Number(match.score ?? 0),
source: String(match.metadata?.source ?? "unknown"),
}));
I ran the same query again.
support-agent (2.1s) ✓
├─ generate-query-embedding (309ms) ✓
├─ embedding-summary (1ms) ✓
│ metadata:
│ dimensions: 1536
├─ tool:retrieve-docs (421ms) ✓
│ metadata:
│ toolName: pinecone.query
│ topK: 5
├─ retrieval-summary (1ms) ✓
│ metadata:
│ matches: 5
│ topScore: 0.89
│ emptyContent: 0
│ sources:
│ - password-reset.md
│ - account-login.md
│ - user-settings.md
│ - security-faq.md
│ - email-delivery.md
├─ build-context (1ms) ✓
├─ context-summary (1ms) ✓
│ metadata:
│ contextChars: 1847
│ contextChunks: 5
└─ llm:gpt-4o (1.3s) ✓
This time, the response was useful:
To reset your password, click “Forgot password” on the login page, enter your account email, and follow the link sent to your inbox.
Same user query. Same retriever. Same model. Different field mapping.
Comparing the Broken and Fixed Runs
The nice part about having both traces locally is that I could compare them instead of relying on memory.
npx agent-inspect diff <broken-run-id> <fixed-run-id> --dir ./.agent-inspect
The important difference was not the final answer. It was the state before the answer.
retrieval-summary
matches: 5
topScore: 0.89
- emptyContent: 5
+ emptyContent: 0
context-summary
- contextChars: 0
- contextChunks: 0
+ contextChars: 1847
+ contextChunks: 5
llm:gpt-4o
- generated with no retrieved context
+ generated with retrieved documentation context
That diff is what I wanted in the first place: evidence that the fix changed the right part of the pipeline.
Not “the answer looks better now.”
A concrete before/after trace showing that retrieval content actually flowed into generation.
Adding a Small Safety Check
After fixing the bug, I added a lightweight check to make this failure mode harder to miss next time.
For this particular route, I wanted a support-answer run to include retrieval before generation and to avoid empty context when retrieval succeeds.
A simple local eval/check command can catch part of that:
npx agent-inspect check .agent-inspect/*.jsonl \
--require-completed \
--detect-stalls
For a project-specific test, I also added an assertion around the metadata I was already recording:
expect(trace.step("retrieval-summary").metadata.matches).toBeGreaterThan(0);
expect(trace.step("retrieval-summary").metadata.emptyContent).toBe(0);
expect(trace.step("context-summary").metadata.contextChars).toBeGreaterThan(0);
That test does not care whether the model says the exact same sentence every time. It checks the deterministic part of the RAG flow:
- Did retrieval return results?
- Did those results contain usable content?
- Did context reach the generation step?
For RAG debugging, that is often more valuable than snapshotting the final answer.
Sharing the Trace Without Sharing the Docs
Because this was a support agent, the trace could have included sensitive metadata if I had been careless.
Before sharing a trace with a teammate or attaching it to an issue, I created a redacted copy and verified it.
npx agent-inspect redact .agent-inspect/*.jsonl \
--profile share \
-o rag-debug.safe.jsonl
npx agent-inspect verify-safe rag-debug.safe.jsonl
That workflow matters. RAG systems often touch private docs, user messages, account names, internal URLs, and vendor-specific metadata. Debug artifacts should not casually become data leaks.
In this case, I did not need raw document text to explain the bug. The safe metadata was enough:
- source file names
- match count
- top score
- number of empty chunks
- context character count
That was the entire diagnosis.
What I Learned
1. Silent Failures Are Common in RAG
The code did not throw. Every step returned successfully.
That is what made the bug annoying.
A retriever can return five matches and still provide zero usable text. A prompt builder can return an empty string and still be “successful.” A model can answer politely from an empty context and still produce a valid response object.
RAG pipelines fail silently all the time because they are mostly glue code.
2. Logs Are Linear, But RAG Is a Tree
My logs said:
Retrieved chunks: 5
Similarity scores: [0.89, 0.85, 0.81, 0.77, 0.73]
Context length: 0
That was technically the same information.
But the execution tree showed the relationship:
retrieve-docs
→ retrieval-summary: matches=5, emptyContent=5
build-context
→ context-summary: contextChars=0
llm:gpt-4o
→ generated with no retrieved context
That structure made the bug obvious.
3. Metadata Is Usually Better Than Raw Content
My first instinct was to dump the chunks.
That would have worked, but it would also create messy and potentially sensitive trace artifacts.
The better debugging metadata was smaller:
await note("retrieval-summary", {
matches: chunks.length,
topScore: chunks[0]?.score,
emptyContent: chunks.filter((chunk) => chunk.content.length === 0).length,
sources: chunks.map((chunk) => chunk.source),
});
I did not need full document bodies. I needed to know whether the document bodies existed.
4. Local Debugging Shortens the Feedback Loop
For production monitoring, I still want a proper observability stack.
For local debugging, I want fast feedback:
npx agent-inspect list
npx agent-inspect view <run-id>
npx agent-inspect report <run-id>
npx agent-inspect diff <broken-run-id> <fixed-run-id>
No hosted dashboard was required for this bug. No trace upload was required. The trace lived in .agent-inspect/, and I could inspect it immediately.
That is the right shape for inner-loop debugging.
When This Workflow Works Best
This approach is especially useful when you have a multi-step AI flow and the final answer is not enough to debug the system.
It helped here because the RAG pipeline had clear internal boundaries:
user question
→ query embedding
→ vector retrieval
→ context building
→ answer generation
The same pattern works for:
- retrieval pipelines
- tool-using agents
- routing logic
- guardrail checks
- multi-agent handoffs
- CI failures where the final assertion does not explain the path
It is not a replacement for production APM or hosted observability. I would still use production-grade monitoring for fleet-wide metrics, retention, dashboards, and alerting.
But for local development, PR artifacts, and debugging one failing run, a local execution tree is hard to beat.
The Five-Minute Setup
To try the same workflow:
npm install agent-inspect
npx agent-inspect init --yes
npx agent-inspect doctor
Wrap the workflow you want to understand:
import { inspectRun, step } from "agent-inspect";
await inspectRun(
"my-rag-agent",
async () => {
const embedding = await step("embed-query", async () => {
// Create query embedding
});
const chunks = await step(
"retrieve-docs",
async () => {
// Call vector database
},
{
type: "tool",
metadata: {
toolName: "vector-search",
},
},
);
await step(
"retrieval-summary",
async () => undefined,
{
type: "state",
metadata: {
matches: chunks.length,
emptyContent: chunks.filter((chunk) => !chunk.content).length,
},
},
);
const answer = await step.llm("generate-answer", async () => {
// Call model
});
return answer;
},
{
traceDir: "./.agent-inspect",
},
);
Then inspect the run:
npx agent-inspect list --dir ./.agent-inspect
npx agent-inspect view <run-id> --dir ./.agent-inspect --verbose
npx agent-inspect report <run-id> --dir ./.agent-inspect
For before/after fixes:
npx agent-inspect diff <broken-run-id> <fixed-run-id> --dir ./.agent-inspect
Before sharing:
npx agent-inspect redact .agent-inspect/*.jsonl \
--profile share \
-o trace.safe.jsonl
npx agent-inspect verify-safe trace.safe.jsonl
Conclusion
The bug was one wrong field name:
- match.metadata.text
+ match.metadata.content
That one line caused the agent to retrieve the right documents, throw away the content, build an empty prompt context, and then answer as if no documentation existed.
The fix took a few minutes. Finding it was the hard part.
The useful shift was moving from “what did the final model say?” to “what happened at each boundary of the pipeline?”
For RAG systems, that boundary-level view is often the difference between guessing and debugging.
Top comments (0)