Visakh Varghese

Posted on Jul 11

RAG Is Not Enough Anymore: The Case for MAG, KAG, and CAG

#ai #machinelearning #rag #systemdesign

RAG was just the beginning - here's how knowledge-augmented generation has fractured into four distinct architectures, and how to pick the right one.

You shipped a RAG pipeline six months ago. It retrieves relevant chunks from a vector store, stuffs them into a prompt, and your LLM returns reasonably accurate answers. It works. And yet - your users keep finding the cracks. The model confidently answers questions that weren't in the retrieved context. It misses multi-hop reasoning. It chokes on structured data. It hallucinates facts that exist in your knowledge base but just weren't in the top-k results.

RAG - Retrieval-Augmented Generation : was a breakthrough when it arrived. The idea was elegant: don't bake all knowledge into model weights; retrieve it at inference time. But as production systems have matured, the community has pushed well past that original formulation. Today, the acronym soup has expanded: MAG, KAG, and CAG are each tackling failure modes that vanilla RAG leaves on the table.

This isn't an academic taxonomy exercise. Each of these architectures makes fundamentally different tradeoffs - between latency and accuracy, between flexibility and structure, between infrastructure complexity and knowledge freshness. If you're building an LLM-powered product that needs to be reliably right, not just impressively fluent, you need to understand the distinction.

RAG: The Baseline You Already Know

Retrieval-Augmented Generation, introduced by Lewis et al. at Meta AI in 2020, follows a deceptively simple pattern: at inference time, encode the user's query, retrieve the most semantically similar documents from an external store, and prepend those documents to the LLM prompt.

Figure 1 : RAG request lifecycle

The mechanism is straightforward, which is why RAG became the default. Vector databases like Pinecone, Weaviate, and Chroma made the infrastructure accessible. Embedding models like text-embedding-3-small made chunk encoding cheap. The entire pipeline fits in a weekend prototype.

Where RAG struggles:

Retrieval ceiling: If the right chunk isn't in top-k, the model has no fallback. You're entirely dependent on embedding quality and chunking strategy.
Multi-hop reasoning: "What did our Q3 report say about the same product line mentioned in the CEO's memo?" - requires chaining two separate retrievals, which flat RAG doesn't do natively.
Structured data: RAG was designed for unstructured text. Ask it to reason over a table of financial figures and you're fighting the architecture.
Context window noise: Irrelevant retrieved chunks actively degrade response quality.

⚠️ Watch out: One of the most common RAG failure modes isn't the LLM hallucinating - it's the retriever silently returning the wrong documents. Always instrument retrieval quality separately from generation quality. They fail in different ways.

MAG: Modality-Augmented Generation

MAG extends RAG's core idea across data modalities. Where RAG assumes your knowledge base is text, MAG says: what if the answer is in a chart, a diagram, a video transcript, or a PDF table?

The key addition is a multimodal retriever and a multimodal LLM (like GPT-4o, Gemini 1.5, or Claude 3.5) that can reason over mixed-modality context.

Figure 2 : MAG architecture: queries fan out across modality stores, results fused into a single LLM context

When MAG makes sense:

Your knowledge base contains PDFs with embedded charts, scanned documents, or slides
Users ask questions that require cross-referencing text and visual data
You're building over financial reports, medical records, engineering specs, or research papers - domains where figures carry critical information

Concrete example: A financial analyst tool where users ask "What was the revenue trend shown in the Q4 slide deck?" - this requires retrieving the actual slide image, not just extracted text around it, because the visual encoding of the trend line contains information the OCR'd text doesn't capture.

Dimension	RAG	MAG
Data modality	Text only	Text + images + tables + audio
Infrastructure complexity	Low	High
LLM requirement	Any capable LLM	Must support multimodal inputs
Cost per query	$0.002–$0.010	$0.020–$0.100
Best for	Text-heavy corpora	Mixed-modality enterprise documents

KAG: Knowledge-Augmented Generation

KAG takes a different fork. Rather than improving what you retrieve, KAG asks: what if we enriched the knowledge representation itself before retrieval?

The core idea is to construct a knowledge graph from your source documents - entities, relationships, properties - and use graph traversal to answer questions rather than (or in addition to) vector similarity search.

Figure 3 : KAG architecture: knowledge graph built offline, traversed at query time for multi-hop reasoning

The retrieval step in KAG isn't similarity search - it's graph traversal. Given "Who approved the budget for Project X and who reports to them?", a knowledge graph can hop Project X -> approved_by -> Amal -> reports_to -> Bibin in one structured traversal - something flat RAG cannot do reliably.

Microsoft's GraphRAG (released 2024) popularised this pattern, showing graph-based retrieval dramatically outperforms flat vector RAG on questions requiring global reasoning across a corpus.

When KAG makes sense:

Your data is naturally relational - org charts, product hierarchies, citation graphs, knowledge ontologies
Users ask multi-hop reasoning questions
You need provenance - being able to trace exactly which facts contributed to an answer, hop by hop
The domain has well-defined entities and relationships (biomedical, legal, financial, technical documentation)

Dimension	RAG	KAG
Knowledge representation	Flat vector chunks	Knowledge graph (entities + edges)
Multi-hop reasoning	Poor	Excellent
Build cost	Low	High (NER pipeline, graph construction)
Query latency	Low (ANN)	Medium (graph traversal)
Knowledge freshness	Upsert chunks	Rebuild or patch graph nodes + edges
Best for	Unstructured document QA	Relational, reasoning-heavy domains

💡 Tip: KAG doesn't mean abandoning vector search. Hybrid KAG systems — graph traversal for multi-hop, vector search for semantic retrieval - outperform either approach alone on benchmarks like HotpotQA and MuSiQue.

CAG: Cache-Augmented Generation

CAG is the newest and most architecturally distinct of the four. Where RAG, MAG, and KAG all augment generation at inference time by retrieving context dynamically, CAG asks: what if we preloaded the entire knowledge base into the model's context and cached the KV state?

This is made practical by modern LLMs with very long context windows (Gemini 1.5 Pro: 1M tokens; Claude 3.5 Sonnet: 200k). Serialize your entire knowledge corpus once, process it through the LLM to build a key-value cache of the attention states, and reuse that cache across all subsequent queries. No retrieval step at all.

Figure 4 - CAG: expensive attention computation runs once offline; every query just loads the cached KV states

The speed advantage is significant. A system that would spend 80-200ms on embedding + ANN search + prompt construction can answer at bare LLM call latency - the context is pre-computed. He et al. (2024) showed CAG matching or exceeding RAG accuracy on static corpora while eliminating retrieval latency and retrieval errors entirely.

When CAG makes sense:

Your knowledge base is static or rarely updated - product manuals, legal contracts, historical records
The corpus fits within the model's context window
You need zero retrieval latency - real-time applications, customer-facing chatbots where every millisecond counts
You want to eliminate the entire class of retrieval failure modes

Where CAG breaks down:

Dynamic data: If your knowledge base changes frequently, rebuilding the KV cache on every update is expensive
Scale ceiling: A 1M token context is roughly 750,000 words - manageable for a focused domain, not for millions of documents
Cost: Preloading and caching large contexts has real compute and memory costs

Dimension	RAG	CAG
Retrieval at inference	Yes (ANN search)	No
Knowledge freshness	Easy (upsert chunks)	Hard (rebuild KV cache)
Corpus size limit	Virtually unlimited	Bounded by context window
Query latency	Medium	Low
Retrieval failure modes	Present	Eliminated
Best fit	Large, dynamic corpora	Static, latency-sensitive apps

📌 Note: CAG's preload step is conceptually similar to a large system prompt with documentation. The difference is that explicit KV caching makes this economically viable at scale by amortising the expensive attention computation across all requests.

Choosing the Right Architecture

The honest answer is that these aren't mutually exclusive - production systems increasingly combine them. But if you're deciding where to invest first:

Start with RAG if you're early-stage. It's the fastest path to a working system and the failure modes are well-understood. Instrument it properly - retrieval quality, context relevance, answer faithfulness - before deciding what to layer on.

Upgrade to MAG when your users' questions require understanding visual or tabular data, and text extraction alone isn't capturing the full picture.

Upgrade to KAG when your retrieval logs show repeated multi-hop failures - questions that require connecting information across multiple documents - or when your domain has strong entity/relationship structure that flat vector search is discarding.

Upgrade to CAG when your knowledge base is bounded and stable, and retrieval latency or retrieval errors are your biggest bottleneck.

The Comparison at a Glance

	RAG	MAG	KAG	CAG
Core idea	Retrieve text chunks	Retrieve across modalities	Graph-based reasoning	Preloaded KV cache
Retrieval mechanism	Vector similarity	Multimodal embedding	Graph traversal	None (cached)
Multi-hop support	Weak	Weak	Strong	Strong (full context)
Modality support	Text	Text + images + tables	Text (+ graph)	Text (bounded)
Knowledge freshness	Easy	Medium	Hard	Hard
Infrastructure cost	Low	High	High	Medium
Query latency	Medium	High	Medium	Low
Best fit	General QA	Mixed-media docs	Relational domains	Static, latency-sensitive

Takeaways

RAG is still the right starting point - treat it as a baseline to benchmark against, not a final destination.
MAG adds modality breadth, not reasoning depth. If the failure mode is "the answer was in an image," MAG is the fix. If the answer was in the text but the retriever missed it, MAG doesn't help.
KAG changes the fundamental knowledge representation. Graph edges encode relationships that embedding similarity cannot. You're building a knowledge engineering pipeline, not just a vector store — the cost is real.
CAG trades flexibility for speed. It eliminates a whole class of failure modes but requires a stable, bounded corpus. It's underutilised today largely because long-context models are still relatively new.
The real-world answer is usually hybrid. KAG for relationship queries, flat RAG for semantic search, CAG for a high-traffic FAQ that never changes.

If you've shipped any of these beyond RAG in production, I'd genuinely like to hear which failure mode pushed you there - and whether the architecture change actually fixed it or just moved the problem somewhere else. Drop it in the comments.

DEV Community