You've built a RAG pipeline. Documents chunked, embeddings generated, vectors in pgvector or Pinecone, retrieval returning solid similarity scores. You spot check it. The right chunks are coming back.
And the answers are still wrong.
Not hallucinated wrong. Not "the model made something up" wrong. Confidently, specifically, plausibly wrong, in a way that takes a week to trace because every component you check looks healthy.
I've debugged enough of these to know where to look first now, and it's almost never the place people start looking.
The gap in the standard pipeline
Here's the architecture everyone ships:
documents → chunk → embed → vector store
↓
query → embed → similarity search → top_k chunks
↓
context = "\n".join(chunks)
↓
prompt = system + context + query
↓
LLM
Look at that context = "\n".join(chunks) line.
That line is doing an enormous amount of unexamined work. It's the entire context layer, compressed into a string join. No ordering logic. No token budget. No permission check. No deduplication. No conflict resolution. No logging.
Every RAG system I've debugged in production had some version of that line, and every serious failure traced back to it.
What actually breaks
Context overflow silently drops your constraints
You're assembling context from multiple sources:
context = (
system_prompt + # ~400 tokens
"\n".join(chunks) + # ~8000 tokens (10 chunks)
format_history(messages) + # ~3000 tokens (4 turns)
tool_output + # ~2000 tokens
business_rules # ~600 tokens
)
# total: ~14,000 tokens
# model context window: 12,000
The API truncates. Usually from the end. Your business_rules were appended last because that felt like the natural order when you wrote it.
They never reach the model. The model answers without the constraint that says "always escalate refunds over $500 to a human." Nobody notices for three weeks, until someone audits the logs.
The fix isn't a bigger context window. It's an explicit budget with a priority order:
BUDGET = {
"business_rules": 800, # never truncated
"system_prompt": 500, # never truncated
"retrieved": 6000, # truncate by rerank score
"history": 2500, # truncate oldest first
"tool_output": 1500, # summarize if over
}
Non-negotiable sources get reserved allocation. Everything else competes for what's left, with a deterministic eviction order you decided in advance, not one the tokenizer decided for you.
Similarity score is not the same as correctness
Two documents in your index:
-
policy_v1.md— updated 2023, deprecated -
policy_v2.md— updated 2025, current
A query comes in. Both are retrieved. policy_v1.md scores 0.87, policy_v2.md scores 0.84, because the user's phrasing happens to echo language from the older doc.
Your top_k sort puts v1 first. The model reads top to bottom, weights early context more heavily, and grounds its answer in the deprecated policy.
Cosine similarity has no concept of "current." It never will. That's not what it measures.
The fix lives in assembly, not retrieval:
def resolve_conflicts(chunks):
by_doc_family = group_by(chunks, key="doc_family")
resolved = []
for family, versions in by_doc_family.items():
if len(versions) > 1:
# newest wins, or include both with explicit annotation
newest = max(versions, key=lambda c: c.meta["updated_at"])
newest.annotation = "CURRENT VERSION"
resolved.append(newest)
else:
resolved.extend(versions)
return resolved
Retrieval surfaces candidates. Assembly decides which ones the model actually sees.
Your vector store has no concept of RBAC
This one is the reason I'd hold a release.
If a chunk is in the index, any semantically similar query can retrieve it. The embedding doesn't know the source document was marked confidential. The similarity search doesn't check user.role.
# what most implementations do
chunks = vector_store.similarity_search(query, k=10)
context = "\n".join(c.text for c in chunks) # 💀
A junior analyst asks a routine question. A chunk from a restricted strategy doc scores above threshold. It goes into context. The model paraphrases it in the answer.
That content is now in a chat log, in whatever logging pipeline you have, possibly in a vector store of conversation history.
The filter has to run between retrieval and assembly:
chunks = vector_store.similarity_search(query, k=30)
chunks = [c for c in chunks if user.can_access(c.meta["acl"])]
chunks = reranker.rank(query, chunks)[:8]
Filter before rerank, not after. Otherwise you burn your top_k budget on chunks the user can't see and end up with three usable results instead of eight.
A model cannot leak what it never received. That's the only guarantee that actually holds.
No instrumentation on the middle layer
You're logging the query. You're logging the response. You might be logging retrieval scores.
Are you logging the assembled context? The actual final string that hit the API?
If not, you can't debug this class of failure. You get a bug report saying "the answer was wrong," you check retrieval (fine), you check the model (fine), and you're stuck.
@dataclass
class ContextTrace:
query_id: str
sources_included: dict[str, int] # source -> token count
sources_truncated: list[str]
chunks_filtered_by_acl: int
chunks_dropped_by_budget: int
final_token_count: int
assembly_version: str
final_context_hash: str
Log this on every request. When something goes wrong, you can reconstruct exactly what the model saw. Without it you're guessing, and guessing about a nondeterministic system is not a debugging strategy.
The layer that should exist
Between retrieval and generation, there should be a module. Not a helper function. A module, with its own tests, its own version, and an owner.
class ContextAssembler:
def __init__(self, budget: TokenBudget, policy: OrderingPolicy):
self.budget = budget
self.policy = policy
def build(
self,
query: str,
retrieved: list[Chunk],
history: list[Message],
tool_outputs: list[ToolResult],
user: User,
) -> tuple[str, ContextTrace]:
retrieved = self._filter_permissions(retrieved, user)
retrieved = self._resolve_conflicts(retrieved)
retrieved = self._deduplicate(retrieved)
sources = {
"rules": self.policy.business_rules,
"retrieved": retrieved,
"tools": tool_outputs,
"history": history,
}
sources, trace = self.budget.apply(sources)
context = self.policy.serialize(sources, query)
return context, trace
The ordering matters more than people expect. Models weight earlier context more heavily. Put constraints first, retrieved content second, history third, the live query last:
[BUSINESS RULES]
...
[RETRIEVED CONTEXT]
<source: policy_v2.md | updated: 2025-03-14 | CURRENT>
...
[CONVERSATION HISTORY]
...
[CURRENT QUERY]
...
Delimited. Labeled. Sourced. Timestamped. The model can distinguish an instruction from a retrieved fact from a previous turn, which it cannot do reliably when you hand it a concatenated blob.
Retrieval improvements that are worth the effort
Since we're here, two things that consistently move the needle before you touch the assembly layer:
Rerank. Retrieve k=30 with vector search, then run a cross-encoder to get down to k=8. Cohere Rerank, bge-reranker-v2-m3, or Jina. Bi-encoder similarity is a fast approximate filter. A cross-encoder actually reads the query and the chunk together. The precision difference is significant and it's maybe forty lines of code.
Structure-aware chunking. RecursiveCharacterTextSplitter with chunk_size=512 will split a markdown table across two chunks and destroy it. For docs with real structure, split on headings and keep tables, code blocks, and numbered procedures atomic. If a chunk exceeds your size limit, that's fine, handle it as an exception rather than mangling the content.
I wrote up the full architecture, including the evaluation setup, in more depth here: Context Engineering for RAG Systems: A Practical Guide for Enterprise AI.
Evaluate the layers separately
Answer-level eval tells you something is wrong. It doesn't tell you where.
# retrieval
- recall@k, precision@k against labeled relevant docs
- hit rate on a curated query set
# context assembly
- % of queries hitting the token ceiling
- % of queries where a non-negotiable source got truncated
- chunks dropped by ACL filter (should be >0 if ACLs work)
- duplicate content rate in final context
- manual review of assembled context for ~30 real queries
# generation
- groundedness (every claim traceable to a context span)
- citation accuracy
- hallucination rate
If retrieval recall is 0.91 and answer accuracy is 0.62, the problem is in assembly. That's a diagnosable gap. Without layer-separated metrics you just have "it's bad sometimes."
Build your eval set from real production queries, not synthetic ones. Users don't ask clean questions. They ask "hey the invoice thing from last week, did that go through?" and your synthetic eval set has nothing like that in it.
This scales badly with agents
Single-turn RAG: one context source.
An agent handling a support escalation:
- CRM lookup (tool call) → account tier, contract status
- RAG retrieval → applicable policy documents
- Conversation memory → last three interactions
- SQL query → current SLA terms, time elapsed
- Business rules → escalation thresholds
- Previous step output → diagnostic result
Six sources, assembled fresh on every step of the loop. Token budget applied across all six. ACL filter applied across all six. Ordering applied across all six. Trace logged for all six.
If you're seeing agents behave unpredictably, check what's in the context window at each step before you touch the orchestration logic. The agent is executing correctly against the input it received. The input is usually the problem.
The thing to take away
context = "\n".join(chunks)
If that line, or something close to it, is in your codebase, that's your context engineering layer. That's the whole thing.
Replace it with a module. Give it a token budget with priorities. Give it a permission filter that runs before rerank. Give it conflict resolution. Give it deterministic ordering. Give it a trace log.
It's not a lot of code. It's maybe two hundred lines and a test suite. It's just work nobody assigns because it doesn't demo well and it doesn't have a framework with a good landing page.
But it's the difference between a RAG system that holds up under real traffic and one that quietly gets deprecated because it was wrong too often and nobody could explain why.
Next time the answer is wrong, log the assembled context and read it. The bug is usually visible in about ten seconds.
Building this out and want a second opinion on the architecture? We do this for a living.
Top comments (0)