How RAG lets an AI answer from your company documents (and what it does not fix)

Adapted from an article originally published in French on closiqode.com.

Retrieval-augmented generation (RAG) lets an AI answer from your own documents. Before writing anything, the system searches your document base for the relevant passages, then builds its answer from them and can say where it came from. The model learns nothing new: at every question, it is handed the excerpts that matter.

The term shows up in every demo and every quote, and it is rarely explained. The business question behind it is simple: how can a model talk about your procedures, technical sheets or terms of sale without ever having seen them? And what do you need to prepare so its answers are right?

RAG in one paragraph

RAG combines two building blocks: a search engine that finds the right passages in your documents, and a language model that writes an answer from those passages. The term comes from a 2020 paper (Lewis et al., arXiv) whose authors already wanted to solve two problems: showing where an answer comes from, and updating a model's knowledge.

The French Directorate General for Enterprise (DGE) sums it up in its November 2024 guide: RAG lets a model "consult" company data before answering. The same guide explains why this is needed: large language models are trained on general text, they know nothing about your company, and their knowledge is frozen at training time.

A useful picture: without RAG, the model answers from memory. With RAG, it answers open book, with the pages the search engine brought in front of it. Answer quality therefore depends as much on the pages retrieved as on the model reading them.

How a RAG pipeline works, step by step

A RAG system works in two phases: document preparation, done once and again at every update, and retrieval plus answer, done at every question.

Before any question: prepare and index

• Conversion: every document becomes text. A scanned PDF goes through OCR, a slide deck is converted to text.
• Cleaning: remove duplicates, repeated headers and footers, cover pages, and set aside documents that contradict each other.
• Chunking: split documents into short segments, from a few sentences to a section. Chunk size is a decisive setting, to be chosen with the people who know the content.
• Enrichment: attach context to every chunk (source document, chapter, date, department) for filtering and citation.
• Indexing: convert every chunk into a vector that represents its meaning, and store those vectors in a vector database.

At every question: retrieve, then answer

• Retrieval: the question is converted too, then compared to the chunks. The system keeps the closest ones by meaning, by shared keywords, or by a combination of both (hybrid search).
• Assembly: the selected chunks go to the model along with the question and instructions such as "answer only from these excerpts".
• Sourced answer: the model writes the answer and lists the documents used. If nothing relevant was found, a well-designed system says so instead of improvising.

In pseudo-code, the query-time half looks roughly like this:

# Pseudo-code, not a specific library
def answer(question, user):
    hits = index.hybrid_search(
        text=question,
        top_k=8,
        filter={"allowed_groups": user.groups},  # access rights at retrieval time
    )
    if not hits or hits[0].score < MIN_SCORE:
        return refuse("I cannot find this in the validated documents.")
    prompt = build_prompt(
        question,
        excerpts=hits,
        rules="Answer only from the excerpts. Cite the source for every claim.",
    )
    return llm.generate(prompt), [h.source for h in hits]

Typical use: a technician asks which setting to apply on a given machine model. The engine finds the right page of the manual, the model phrases the answer and points to the manual. If that manual is not in the base, the correct answer is "I cannot find this information", not a plausible value.

RAG versus a model trained on your data

RAG does not train the model on your documents. It shows them to the model at question time, and the model does not keep them. This is the most common confusion, and it has practical consequences for updates, confidentiality and traceability.

Fine-tuning changes the model's internal parameters. The knowledge gets blended in: you cannot tell which document a sentence came from, or cleanly remove an outdated fact. The French cybersecurity agency (ANSSI) notes in its April 2024 security recommendations for generative AI that user access rights cannot be managed on training data, whereas they can be on data added in production, such as a vector database.

Criterion	RAG	Fine-tuned model
Where the knowledge lives	In your documents, consulted at every question	In the model's parameters
Updating	Replace the document and reindex it	A new training run
Citing the source	Possible, passage by passage	No reliable link to a document
Access rights	Filterable per user	Not manageable, according to ANSSI
Good fit	Answering from a corpus that changes	Enforcing a very specific style, format or vocabulary

For a small or mid-sized company that wants to query its procedures, sheets or contracts, RAG is the logical starting point. Fine-tuning is rarely justified at that scale, and it fits poorly with information that changes often.

Which documents work, and which cause trouble

RAG performs well on written, current, non-contradictory text. It struggles with complex tables, poor scans and questions that require reading the whole corpus. The DGE guide notes that RAG is generally better on text than on tables, and less suited to summaries spanning many documents.

Document type	What happens	What to plan
Procedures, internal guides, written FAQs	Good fit: each passage stands on its own	Make sure only one version circulates
Technical sheets, manuals	Good fit if the text is machine-readable	Keep product references in the text, not only in an image
Terms and conditions, contract templates	Good for finding a clause	Date every version and remove old ones
Price or specification tables	Risky: chunking can separate a value from its header	Specific processing or a structured source
Old scans, handwritten documents	Depends on OCR quality	Check the extracted text before indexing
Email threads	Noisy: repetition, provisional opinions, decisions later reversed	Move final decisions into a reference document

A good starting scope is often a small set of reference documents: the ones that answer the questions asked every week.

The limits to know before you start

RAG reduces fabricated answers without eliminating them, and it is never better than the documents you give it. The DGE guide says explicitly that hallucinations can still occur with RAG. Four limits come up in practice:

• Retrieval misses: only a limited number of chunks are kept. If the right information is phrased differently or spread across documents, it may not come up.
• Documents contradict each other: two versions of a procedure give two possible answers, and the model may pick the wrong one or blend both.
• Outdated versions stay indexed: a forgotten old price list keeps producing wrong answers, with a source to back them.
• A citation does not prove everything: you still have to check that the cited passage actually supports the generated sentence.

Preparing your documents

Preparation matters as much as the technology. The DGE guide calls it an essential step and stresses that domain expertise remains indispensable, even when a contractor handles it.

• List real questions for a few weeks, with the expected answer and the document that contains it.
• Designate reference documents: one dated version per topic, with an owner.
• Remove duplicates and old versions, or archive them outside the queried scope.
• Spot what is written nowhere: answers that only exist in someone's head must be written down first.
• Classify by access level.
• Check for personal data: the DGE guide lists anonymization or pseudonymization among the cleaning steps.

The rule I set before developing anything: a useful company RAG answers from a validated corpus, cites the document used, and refuses explicitly when the base does not contain the information. It is tested with real questions before going live, including the ones it must refuse.

FAQ

Does RAG train the model on my documents?
No. Documents stay in a separate base and only a few excerpts are sent per question. Still check the model provider's terms: ANSSI notes that prompts and answers are sometimes reused for training.

How do you update it when a document changes?
Replace the document at the source, reindex it, remove the old version from the index, then replay the test questions.

Open book, not a better memory

RAG gives the model the right pages. Choosing, cleaning and testing those pages is still the real work.

---

I am Fabien Kost, an independent developer in France (ClosiQode). I build business AI assistants that answer from validated documents, cite their sources and say so when they cannot find the answer. More on how I design them (in French).

Sources: DGE, guide to retrieval-augmented generation (in French, PDF); ANSSI, security recommendations for a generative AI system (in French); Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv, 2020).