One of the biggest surprises I had while working on RAG systems wasn't retrieval.
It was what happened after retrieval.
Most conversations around RAG optimization focus on:
- Better embeddings
- Better rerankers
- Better chunking
- Better vector databases
All important.
But here's something I kept noticing.
Every query forces the model to reconstruct the same understanding from scratch.
Imagine your pipeline looks like this:
User Question
↓
Retrieve Documents
↓
LLM reads everything
↓
Builds understanding
↓
Generates answer
Even if another user asks a nearly identical question five minutes later, the model repeats almost all of that expensive reasoning.
Not because it has to.
Because we've mostly been caching outputs, not understanding.
The expensive part isn't always generation
Generation usually gets the attention because it's visible.
The hidden cost is rebuilding context.
The model has to:
- Connect documents
- Identify relationships
- Discard irrelevant facts
- Resolve contradictions
- Build a coherent understanding before it can answer
That work gets thrown away after every request.
What if we cached the synthesized context?
Instead of caching the final answer, cache the structured understanding that the model already built.
That allows future requests to reuse reasoning while still generating fresh answers from up-to-date data.
The benefit is twofold:
- Fewer tokens
- Lower latency
without serving stale responses.
This also changes how we think about RAG
Instead of optimizing only retrieval, we start optimizing context construction itself.
Retrieval finds information.
Context construction creates understanding.
Those aren't the same problem.
This line of thinking is what eventually led us to build Coalent.
Coalent is an open-source context engineering library that focuses on reusing synthesized context instead of repeatedly reconstructing it from the same retrieved documents.
Rather than caching answers, the goal is to preserve the expensive understanding that happens between retrieval and generation—while still allowing responses to stay grounded in the latest source data.
As AI applications become more agentic, I think this distinction will matter more and more.
I'm curious how others are approaching this.
Are you caching answers, retrieved chunks, synthesized context, or something else entirely?
If you're interested in exploring this approach,
Coalent is open source: https://github.com/Vectorlink-Labs/coalent
I'd genuinely love to hear feedback from anyone experimenting with context engineering, RAG optimization or AI agent memory. Different perspectives are always valuable and I'm curious how others are tackling the same challenges.
Top comments (0)