If your app still treats context as embed everything and hope, you are stacking up the failure modes these guests describe. These five Chain of Thought conversations on retrieval and context all point at the same job: decide what the model sees, how it gets there, and what happens on the next turn.
Where does vector-only RAG stop being enough?
In Hallucinations Are a Data Architecture Problem, Neo4j's Sudhir Hasbe walks the familiar pipeline end to end: chunk, embed, similarity search. That is the standard pipeline he describes. His argument is that many model hallucinations are really missing or wrong enterprise context. LLMs were not trained on your internal knowledge, so dumping a disconnected data lake into retrieval does not give the model relationships to reason over.
The practical move I take from that episode: keep vectors for candidate finding, then add structure on top. Hasbe's framing combines vector retrieval with graph relationships and metadata filters (GraphRAG in his vocabulary). Before you blame the LLM, sketch what entities and edges your answer depends on, and ask whether similarity alone can surface them. If the question needs hops across systems or policy constraints, plan graph traversal or a semantic map before you widen top-k.
What should agent memory optimize for?
Agent Memory: The Last Battleground in the AI Stack features Richmond Alake, who has been unusually concrete about memory in production. His line from building at Oracle is that vector search was a key component but not the whole memory layer. Retrieval strategy is what the rest sits on.
For builders, that means naming memory as its own design problem, separate from prompt tweaks. Alake maps working, episodic, semantic, and procedural memory to agent architecture, and he warns about two prototype traps: the wrong mental model, and hard-deleting memories instead of controlled forgetting. I'd start by writing down which memories must survive sessions, which should decay, and which retrieval modes (vector, graph, relational, spatial in his demo story) you need in one user turn. Files are fine for demos; production memory usually wants a database and explicit forget rules.
Does the frontier model deserve credit for your RAG wins?
Fergal Reid's How Intercom Cut $250K/Month by Ditching GPT for Open Models is a cost story on the surface and a context story underneath. Intercom was spending $250K per month on one summarization workload on GPT before moving to a fine-tuned open model. Reid also describes what he believes was among the first production RAG deployments on GPT-4 launch day, and years of iterating chunking and the context pipeline behind Fin.
The lesson I would steal for any team: measure resolution and cost, but invest in the surrounding retrieval stack. Reid attributes much of Fin's resolution improvement to custom re-rankers, retrieval models, and query canonicalization rather than to swapping the biggest chat model. Context engineering here means owning canonicalization, chunk boundaries, and reranking as first-class code paths. Swap the LLM only after those paths are visible in metrics.
Who owns the retrieval half nobody demos?
Maxime Labonne on Beyond Transformers: How Liquid AI Is Rethinking LLM Architecture spends time on the unglamorous side of RAG: embedding models themselves. Liquid AI's LFM2 line is a hybrid stack (six attention layers and ten short convolution layers in his description, more convolution than attention). For retrieval specifically, he explains why they built a late-interaction ColBERT-style retriever instead of defaulting to classic single-vector embeddings.
If you only benchmark the generator, you cannot see the retrieval half. I'd add a retrieval eval that matches your query distribution, and compare single-vector versus late-interaction style scoring when answers need token-level overlap or fine distinctions. Architecture choices upstream of the prompt change what context is even eligible to be retrieved.
How does RAG change when the model picks the next fetch?
Bob van Luijt's AI Infrastructure & the Evolution of RAG goes back to when Weaviate called the pattern "generative search" before "RAG" was universal. Classic RAG is one direction: query, retrieve, generate. Agents flip that into a feedback loop where the model decides what to fetch next.
Design for that loop early. Your context layer needs stable IDs, filterable metadata, and APIs the agent can call more than once per user message. One-shot retrieval pipelines break the moment the agent needs to refine a search or pull a second source. Treat each fetch as a state transition, log it, and cap tool fan-out.
Checklist
- Map the question to entities and relationships before you widen
top_k; add graph or metadata constraints when similarity alone cannot connect the dots. - Treat agent memory as retrieval strategy: typed stores, session rules, and forget/decay instead of delete.
- Ship and measure canonicalization, chunking, and reranking before you assume a larger LLM will fix bad context.
- Benchmark and choose embedding and retriever designs (single-vector vs late-interaction) on your own queries.
- Build retrieval APIs for repeated, model-driven fetches in agent loops as well as a single retrieve-then-generate pass.
More on this topic, with the related episodes, is on Chain of Thought. It draws on this episode.
Subscribe to the Chain of Thought newsletter for new episodes and write-ups like this one.
Drafted with AI assistance from the episode transcripts.
Top comments (1)
"Sketch what entities and edges your answer depends on before you widen top-k" is the most practical line here. Raising top-k usually buys more noise, not the missing relationship. Did any guest have a rule for when graph traversal is worth the extra latency versus a simple metadata filter?