Retrieval-augmented generation (RAG) is a pattern where an AI assistant searches your own documents first, then writes its answer from the passages it found and cites them. The model is not trained on your files. Mickai builds these private knowledge bases on hardware you own, so the documents stay on your own estate.
What does retrieval-augmented generation actually mean?
RAG means the assistant looks something up before it answers. There are two halves. Retrieval: a search runs across a collection of documents you control, and returns the passages that bear on the question. Generation: a language model reads those passages and writes an answer out of them.
The term comes from a 2020 research paper by Lewis and colleagues (arXiv:2005.11401), which set out the pattern of combining a retrieval step with a generative model so the system could draw on a body of knowledge it had not been trained on. It has since become the ordinary way to put an assistant in front of an organisation's own material.
The plainest version I can give you: instead of asking a model what it remembers, you ask it to read.
How does RAG work, step by step?
Four steps, in order. First, indexing. Your documents are broken into passages, and each passage is stored alongside a mathematical representation of its meaning, in a local index. Nothing about the model changes during this step. You are building a searchable map of your own material.
Second, retrieval. A question arrives. The system converts it the same way and finds the passages that sit closest to it, usually combined with ordinary keyword matching so that exact terms like a policy number or a clause reference are not lost.
Third, assembly. The winning passages are placed in front of the model together with the question and an instruction along the lines of: answer using only what follows, and say so if it is not there.
Fourth, generation. The model writes the answer, and each claim is tied back to the passage it came from. In our builds, an assistant that cannot support a sentence with a retrieved passage is expected to say it does not know. An assistant that fills the gap instead is the one that gets you into trouble.
The whole loop runs at question time. There is no training run and no waiting period between adding a document and being able to ask about it. One caveat on our own build: a local OCR runtime has read scanned PDFs in controlled tests, and the SIOS extraction and ingestion integration is still being completed, so treat scanned-document handling as work in progress rather than a shipped feature.
Does RAG train an AI model on our company documents?
No. This is the part most buyers get wrong, and it is worth being exact about.
The model's weights are fixed. Your documents go into an index, which sits next to the model, and passages are handed to it for the duration of a single answer. When the answer is finished, those passages are gone from the model's working memory. Nothing about your files has been absorbed into the model, so nothing about your files can resurface in an answer given to somebody else.
That distinction decides how a data protection review goes. Under UK GDPR you need to know where personal data sits, who can reach it, and how to remove it. An index is a store you can point at, audit, restrict by permission, and delete from. Knowledge folded into a set of weights is none of those things. The ICO's guidance on AI and data protection is the right starting point for framing that assessment with your data protection officer.
What is the difference between RAG and fine-tuning?
They solve different problems, and they get sold as alternatives when they are not.
Fine-tuning adjusts a model's weights so it behaves differently: a house tone of voice, a particular output format, a specialist vocabulary it keeps mangling. It teaches style and shape.
RAG supplies facts. It gives the model access to material it has never seen, at the moment the question is asked.
If your problem is that the assistant does not know your procurement policy, fine-tuning is the wrong instrument. Training documents into weights buries them. You cannot cite them, you cannot see which one was used, and you cannot remove one without retraining. If your problem is that it answers correctly but writes like a brochure, retrieval will not fix that.
Most serious internal deployments end up doing both: retrieval for knowledge, a specialised model for behaviour. In our system those specialised models are what we call brains. There are fifty of them, each tuned for a particular kind of work, retrieving from knowledge bases the customer owns.
Why do regulated organisations prefer RAG for internal knowledge?
Because it is the only arrangement where you can answer the auditor's question. Not "is the AI accurate", which nobody can promise, but "where did that answer come from, and who signed it off".
Four properties do the work. Provenance: the answer arrives with its sources, so a reviewer checks the passage instead of trusting the summary. Currency: correcting the document corrects the assistant, straight away. Access control: retrieval can respect the permissions the document already carries, so a contractor's question does not surface a board paper. Removal: deleting a document and reindexing puts it beyond the assistant's reach, which is what a retention schedule or a subject access request actually requires.
The NCSC's guidelines for secure AI system development treat an AI system as something to be secured across its whole life, from design through development and deployment to operation and maintenance. That is a sensible frame for a procurement conversation, and a retrieval system built on an index you hold is easier to reason about under it than one whose knowledge is opaque.
None of this is an argument against cloud. Cloud remains a good place for work that is not regulated and data that is not sensitive. The assumption I do argue with is the one that says a regulated organisation has no option but to rent its intelligence, ship its records offsite, and take a vendor's word for what happened to them.
What can go wrong with RAG, and how do we check it?
Plenty, and anyone telling you otherwise has not run one in anger.
Retrieval misses: the right passage exists, but the search does not surface it, often because the question uses the business's words and the document uses the regulator's. The index goes stale, and the assistant quotes a superseded procedure with total composure. Passages get split badly, so a condition arrives without the exception that qualifies it. Permissions are applied to the document library but not to the index. And the failure that actually costs money: the assistant summarises a retrieved passage fluently while getting its sense wrong.
Retrieval reduces invention. It does not abolish it.
So we build for inspection rather than for reassurance. Every answer shows its sources, and the reviewer is expected to open one. Consequential actions wait for a named person to approve them: the assistant drafts, a human decides. And every consequential action is sealed into the Open Audit Record under ML-DSA-65, the post-quantum signature scheme NIST published as FIPS 204 in 2024, so an auditor can export the record and verify it offline with a public key, using tools that are not ours.
That record is tamper-evident. It is not the other thing, and the difference is the whole point. Nothing stops somebody altering a stored record. What the signature does is make the alteration fail verification, so you find out. Any vendor promising you logs that cannot be altered is promising something cryptography does not offer.
Can RAG run entirely offline on our own servers?
Yes, and for a lot of buyers that is the only version worth discussing.
The Mickai Sovereign Intelligence Operating System installs on hardware the customer owns, in three shapes: a single workstation for a team, a server in your own rack for a department, and a fully air-gapped install with no route out at all. The models, the index and the audit record all sit inside your perimeter. There is no data egress, because there is nowhere for the data to go.
The platform ships 63 studios in total: 14 production-ready at launch and 49 in development. Private knowledge bases sit in the production-ready group, with the extraction and ingestion integration still being completed. There is more on internal knowledge AI and on what sovereign AI means in practice, and a longer piece on on-premise RAG and knowledge sovereignty.
Our closed beta is open, with one regulated company onboarding as a design partner. If you are weighing this up for an organisation where the answer has to be defensible as well as quick, that is the conversation worth having.
Frequently asked questions
Is RAG the same thing as an AI search engine?
No. A search engine returns a ranked list of documents and leaves the reading to you. RAG runs that search, then has a language model read the top passages and write a direct answer with its sources attached. Search finds the material. Retrieval-augmented generation finds it and then answers the question you actually asked.
Delete this FAQ entry in full, question and answer, leaving five self-contained questions. It duplicates the H2 "Does RAG train an AI model on our company documents?", which already answers it in the body.
No. Training is not part of the pattern. Your documents are indexed, which builds a searchable map of them, and the relevant passages are passed to the model only while it answers a single question. The model's weights never change, so your material cannot end up inside the model or in somebody else's answer.
Can I see which document an answer came from?
Yes, and you should insist on it. A properly built RAG assistant cites the passage behind each claim, so a reviewer can open the source and check it rather than trusting the summary. In our system, consequential actions are sealed into the Open Audit Record, which records the action and the named person who approved it, so a reviewer can verify the entry offline.
Does RAG work without an internet connection?
It can, if it is built to. Retrieval and generation are both local computations, so nothing about the pattern itself needs the internet. What matters is where the software runs. SIOS installs on hardware the customer owns, including a fully air-gapped configuration with no outbound route, where there is no egress to worry about.
What happens if we delete a document from the knowledge base?
Delete the document, reindex, and it is beyond the assistant's reach. The passages leave the index with the file, so the next question cannot retrieve them. That is precisely why the pattern suits retention schedules and subject access requests, and why training documents into model weights does not: weights cannot be edited one record at a time.
Is RAG safe for confidential or privileged documents?
It can be, on two conditions. The retrieval layer must enforce the same permissions as the document store, so privilege is not lost at the index. And the whole system must run somewhere you control, because confidentiality survives no further than your perimeter. Put the index on somebody else's estate and you have moved the problem, not solved it.
Written by Micky Irons, founder and chief executive of Mickai LTD, which builds a sovereign AI operating system for regulated organisations. More at mickai.co.uk.
Top comments (0)