What Is RAG? Retrieval-Augmented Generation Explained

Originally published on DevToolHub.

What is RAG, in one sentence? A way to make an LLM answer questions using documents it was never trained on, by searching those documents for relevant passages and handing the model the results as context before it generates a response. No retraining, no fine-tuning — just a search step bolted onto the front of a normal prompt.

The term comes from a 2020 Facebook AI Research paper that described it as combining "pre-trained parametric and non-parametric memory for language generation" — the model's trained knowledge, plus a live lookup into an external index.

What Is RAG?

Retrieval-augmented generation is a pattern, not a specific tool: retrieve relevant information for a query, then generate an answer using that information as context. The original RAG paper, authored by Patrick Lewis and colleagues at Facebook AI Research, paired a pre-trained sequence-to-sequence model with a dense vector index accessed by a pre-trained neural retriever.

RAG exists because LLMs have two hard limits. Their knowledge is frozen at training time, and they don't know your private data at all. Retrieval solves both by pulling relevant text in at request time instead of baking it into the weights.

What Is RAG Actually Doing When It Retrieves Data?

The pipeline runs a fixed sequence for every question:

1. Chunk your documents into smaller passages
2. Embed each chunk into a vector using an embedding model
3. Store the vectors in a vector database
4. Embed the incoming question the same way
5. Retrieve the chunks whose vectors are closest to the question
6. Generate an answer using those chunks as context

Any LLM can handle the generation step. The retrieval half is what makes RAG RAG.

What Is a Vector Database, and Why Does RAG Need One?

A vector database indexes by geometric closeness rather than exact values, so a search for "how do I reset a password" can match "forgot your login credentials" even with almost no word overlap. Ollama's API exposes /api/embed for this: send {"model": "...", "input": "..."}, get back an array of floats per input. nomic-embed-text is a solid local default — Ollama's own listing states it beats OpenAI's text-embedding-ada-002 and text-embedding-3-small on short and long context tasks, at a 274MB download.

Why More Retrieved Chunks Doesn't Mean Better Answers

A 2023 Stanford/Berkeley/Samaya AI study found model performance "is often highest when relevant information occurs at the beginning or end of the input context" and "significantly degrades" when it's buried in the middle — the "lost in the middle" effect. A tighter retrieval step of 3-5 genuinely relevant, well-ranked chunks tends to outperform a looser one with more, worse-ranked results.

RAG vs Fine-Tuning

RAG adds new knowledge without touching the model's weights — good for data that changes. Fine-tuning changes model behavior, tone, or output format by retraining the weights. They solve different problems and plenty of production systems use both.

Common Mistakes

Chunking without testing retrieval quality. Assuming the retriever always finds something relevant (it doesn't stop hallucination, it just changes what gets hallucinated on top of). Skipping re-indexing after data changes. Treating every task as a RAG problem when the answer already lives in the model's training data.

Full article with the FAQ and quick summary: devtoolhub.com/what-is-rag