Ask a large language model for a specific statistic, then ask where it found that number. More often than not, the citation it gives you doesn't exist. The model will hallucinate a plausible-looking reference, confidently present outdated conclusions, or simply make things up without any internal signal that something is wrong. This failure mode has a well-known name — hallucination — and the most widely adopted engineering solution for it is RAG.
RAG in One Sentence
RAG stands for Retrieval-Augmented Generation. The idea is straightforward: before the LLM generates an answer, retrieve relevant document chunks from an external knowledge base, then feed those chunks to the model as context so it can compose its response based on real source material rather than parametric memory alone.
Think of it like writing a research paper. You don't cite statistics from memory; you look them up first, then write your argument around verified data. RAG gives language models the same "look it up, then write" workflow.
Three Structural Limitations of LLMs
To understand why RAG is necessary, we need to identify the specific gaps it fills.
Knowledge cutoff. Every model has a training data deadline. GPT-4's cutoff is late 2023; Claude's is early 2025. Anything that happened after that deadline simply doesn't exist in the model's world. It will either admit ignorance or, more dangerously, fabricate an answer that sounds current.
Bounded parametric capacity. Even a 100-billion-parameter model can only "memorize" so much. Long-tail facts, niche domain knowledge, your company's internal documentation, yesterday's meeting notes — none of these are in the weights.
No built-in fact-checking. Token generation is probabilistic sampling. The model has no mechanism to distinguish whether it's recalling a training fact or pattern-matching its way into a plausible-sounding fiction.
RAG addresses all three: it supplies up-to-date, verifiable, externally sourced evidence at inference time.
How RAG Works: Three Stages
A standard RAG pipeline has three phases — Indexing, Retrieval, and Generation.
Indexing transforms your raw knowledge base into a searchable format. Documents are split into chunks (typically 512–1024 tokens), each chunk is converted into a high-dimensional vector using an embedding model, and those vectors are stored in a vector database (FAISS, Milvus, Chroma, Pinecone, etc.).
Retrieval finds the most relevant chunks for a given query. The user's question is also embedded into the same vector space, then a similarity search (cosine similarity or dot product) returns the Top-K closest chunks. Advanced pipelines add a reranking step using a cross-encoder to refine initial results.
Generation concatenates the retrieved chunks with the original question into a prompt, then sends it to the LLM. The model's job shifts from "answer from memory" to "answer based on the provided material" — essentially an open-book exam.
Embeddings: The Foundation of Retrieval Quality
Retrieval quality depends heavily on the embedding model. An embedding compresses a text passage into a fixed-length numerical vector (commonly 768 or 1024 dimensions) such that semantically similar texts end up close together in vector space.
For example, "how to improve code quality" and "writing better code" should map to nearby vectors, while "nice weather today" should land in a completely different region.
Popular choices include OpenAI's text-embedding-3, BGE, and E5 families. When evaluating options, look at retrieval task scores on the MTEB leaderboard rather than raw parameter counts.
Vector Database Selection
Vector databases differ from relational databases in a fundamental way: relational DBs excel at exact matching ("find record id=123"), while vector DBs excel at approximate nearest-neighbor search ("find the 10 passages most semantically similar to this query").
Quick selection guide:
- FAISS — high-performance single-node scenarios
- Milvus — distributed, large-scale deployments
- Chroma — lightweight prototyping
- Pinecone — fully managed cloud service
Consider your data scale, latency requirements, persistence needs, and operational capacity.
Chunking Strategy Matters More Than You Think
Chunk too large and you dilute relevance with noise. Chunk too small and you lose surrounding context, making retrieved passages hard to interpret.
Common strategies include fixed-length chunking (by token count), semantic chunking (split at paragraph or section boundaries), and recursive chunking (split by large structures first, then sub-divide). Start with 512–1024 tokens per chunk and adjust based on your specific document types and downstream evaluation metrics.
RAG vs Fine-tuning: When to Use Which
Both RAG and fine-tuning help models "learn" new knowledge, but they serve different purposes.
RAG works best when:
- Knowledge changes frequently (just update the vector store, no retraining)
- You need citations and source attribution
- You want to augment multiple model versions with the same knowledge base
Fine-tuning works best when:
- You need to change the model's output style or reasoning pattern
- You have stable, domain-specific reasoning chains to internalize
- Low-latency inference matters and you can't afford retrieval overhead
In practice, they're not mutually exclusive. Many production systems use RAG for knowledge grounding and fine-tuning for behavioral alignment.
RAG for Local Models: Privacy Meets Capability
RAG's value becomes even more pronounced for models running on edge devices. Local models typically sit in the 4B–8B parameter range, which means their "memory capacity" is inherently limited. At the same time, the core motivation for local deployment is usually data privacy — keeping sensitive information off cloud servers.
Local model + local RAG gives you the best of both worlds: high-quality answers grounded in external knowledge, with all data — documents and queries alike — staying on-device.
Our team has been working on edge AI for a while. Mano-P is an open-source GUI agent model built for Apple Silicon devices. Its 4B quantized version runs locally on a Mac mini (M4 chip, 32GB RAM) at ~80 tokens/s decode speed, and the companion Cider SDK adds INT8 activation quantization that delivers 1.4x–2.2x prefill speedup. Mano-P currently ranks #1 on OSWorld among specialized GUI agent models with a 58.2% success rate, with all inference executed entirely on-device — screenshots and task data never leave the machine.
Getting Started
For developers looking to build their first RAG pipeline, the fastest path is:
- Pick a framework (LangChain or LlamaIndex)
- Load your documents and configure chunking
- Choose an embedding model and vector store
- Wire up retrieval and generation
- Build an evaluation set and iterate on each component
The engineering complexity of RAG is manageable. What takes time is systematic optimization — measuring retrieval precision, tuning chunk sizes, experimenting with reranking, and crafting prompt templates that help the model make good use of retrieved context.
If you're exploring local AI agents or edge-native inference, check out Mano-P on GitHub. Stars are always appreciated ⭐
Top comments (0)