Most health organisations have somewhere between a few hundred and a few thousand internal documents that tell staff how to do things. Consent forms. Incident escalation. Rostering rules. Infection control procedures. Which form goes to which committee, and by when. The documents exist, they are mostly current, and almost nobody can find the right one in under five minutes. So people ring the ward clerk, or the quality manager, or whoever has been there longest, and that person answers the same twelve questions every week.
This is the least glamorous AI use case in health and probably the best first one. It sits entirely on the administrative side, the source material is text you already own and control, and when it goes wrong the failure is visible rather than silent. This post is part of our Practical AI in Health series, and it covers what we actually build for this: retrieval-augmented generation over an internal document set, with citations, hard refusals, and an evaluation you run before anyone outside the project team touches it.
Scope it narrowly, on purpose
The first design decision is what the assistant will not answer, and it is a compliance decision rather than a product one.
The TGA's guidance on clinical decision support software sets a three-part exemption test in Schedule 4 Part 2 of the Therapeutic Goods (Medical Devices) Regulations 2002, in force since 25 February 2021. Software has to only support recommendations to a health professional, must not process medical images or signals from other devices, and must not replace clinical judgment. All three have to be met, and the TGA is explicit that calling something "decision support" does not make it exempt. Even where the exemption applies, sponsors must notify the TGA within 30 working days of supply, meet the Essential Principles, and report adverse events.
A chatbot that answers "what is our procedure for documenting a medication error" is an administrative tool. The same chatbot, if it starts answering "what dose should I give," has moved into a different regulatory conversation. So the refusal behaviour is not politeness. It is the control that keeps the system inside the scope you scoped it for. Where an answer touches anything clinically adjacent, the design requirement is that a named human signs off before the answer is acted on, and the interface says so at the point of use rather than in a footer.
On the broader regulatory picture, the Commonwealth's Safe and Responsible AI in Health Care review (final report, 2025, drawing on 69 written submissions) concluded that existing health portfolio legislation can largely accommodate AI with minor and technical amendments. There is no new AI act to comply with. Your obligations are the ones you already have around privacy, records and clinical governance.
The architecture
The shape of the system is unremarkable, which is the point:
- Ingestion. Pull documents from wherever they live: SharePoint, the intranet, a policy management system. Keep the document ID, version, owner, approval date and review date as metadata on every chunk.
- Chunking. Split on document structure, not fixed token counts. Policy documents have numbered clauses and headings, and those boundaries are what a person will want to be pointed at.
- Embedding and index. A vector store with metadata filters, so you can restrict retrieval to current versions and to the document sets a given role is allowed to see.
- Retrieval. Hybrid search (dense vectors plus keyword) usually beats either alone on policy text, because staff use exact form numbers and internal acronyms.
- Generation. A prompt that instructs the model to answer only from retrieved chunks, to quote the clause, and to say it does not know when the retrieved text does not contain the answer.
- Citation rendering. Every claim in the answer links to a document, a version and a section, so the reader can open the source in one click.
- Logging. Question, retrieved chunk IDs, answer, citations, user feedback. This is your audit trail and your evaluation data.
A chunk record looks roughly like this:
{
"chunk_id": "POL-CG-014-v3.2#4.1.2",
"doc_id": "POL-CG-014",
"title": "Open Disclosure Policy",
"version": "3.2",
"approved": "2025-11-04",
"review_due": "2027-11-04",
"section": "4.1.2 Notifying the patient",
"audience": ["clinical", "admin"],
"text": "..."
}
Carrying review_due through to the answer matters more than it looks. An assistant that confidently quotes a policy which expired eight months ago is worse than a search box, because it has removed the moment where the reader would have noticed the date on the cover page.
Why RAG rather than fine-tuning
We get asked this on nearly every engagement, usually phrased as "can we just train the model on our policies."
Ovadia et al. tested exactly that comparison at EMNLP 2024, putting a base model, unsupervised fine-tuning, RAG, and fine-tuning plus RAG against knowledge-intensive tasks. RAG consistently outperformed fine-tuning for both existing and entirely new knowledge, and combining the two did not reliably beat RAG on its own.
The practical arguments are stronger than the benchmark one. Fine-tuning bakes knowledge into weights, so it cannot tell you which document an answer came from, and you cannot cite what you cannot locate. When a policy is revised, RAG needs a re-index and fine-tuning needs a retraining cycle. And when a policy is withdrawn, RAG deletes the chunk while fine-tuning leaves the old text somewhere in the weights with no reliable way to remove it.
Retrieval quality is where accuracy is won. A published RAG chatbot over hospital EMR manuals, the closest analogue to an internal policy assistant we have seen in the literature, built a 5,931 question-document evaluation set and raised top-k retrieval accuracy to 97.6% by fine-tuning the embedding model rather than changing the LLM.
Citations are the safety mechanism, and they can also lie
"We use RAG" is not a safety claim. Stanford RegLab's preregistered study of commercial legal research tools, published in the Journal of Empirical Legal Studies in 2025, hand-scored 202 queries and found hallucination rates of roughly 17% for Lexis+ AI and 33% for Westlaw AI-Assisted Research, against 43% for GPT-4. RAG helped. It did not eliminate the problem, despite vendors advertising "hallucination-free" citations.
The finding that should change your design is the one about citation hallucination: an answer that cites a real but wrong source, which the authors argue may be even more pernicious than outright invention, because the presence of a reference is what makes reviewers stop checking. That is the same trap the Australian Commission on Safety and Quality in Health Care describes as automation bias, with errors of commission (acting on an incorrect recommendation) and errors of omission (failing to act when the tool misses something).
So we score citation correctness separately from answer correctness, and we show the quoted clause text inline rather than a bare reference number. If the reader can see the sentence the answer came from, checking costs three seconds instead of three minutes.
Evaluation before rollout
Do not launch on vibes and a demo to the executive team. The evaluation design we use follows a published 2025 RAG deployment: build a test set of around 100 questions with ground-truth answers reviewed by two subject matter experts, then score with RAGAS metrics that separate retrieval failure (context precision, context recall) from generation failure (faithfulness, answer relevancy).
The separation is what makes the numbers actionable. Low faithfulness with good retrieval means the model is inventing and you fix the prompt or the model. Low context recall means the policy was never retrieved and you fix chunking, embeddings or the index. Add two categories your test set will not otherwise cover: out-of-scope questions that must be refused, and questions whose answer sits in a superseded document, where the correct behaviour is to cite the current version.
The Commission's guide is structured around "before you use", "while you use" and "after you use", which maps neatly onto a rollout plan: evaluation gate, supervised pilot with feedback capture, then periodic re-evaluation as documents change.
Privacy, hosting and cost
The OAIC's October 2024 guidance treats any organisation using a commercial AI product, including purely internally, as a "deployer" with obligations under the Privacy Act 1988 and the 13 Australian Privacy Principles. Policy documents are usually not personal information, but the query logs can be, because "what do I do if I made this error" is a question about a person.
That argues for keeping the index and the logs in Australian-hosted, access-controlled infrastructure, with data-retention terms in the model provider contract that prohibit training on your traffic. The risk context is not abstract: health service providers were the most-breached sector in Australia in calendar 2025, with 225 of 1,205 notifications, ahead of finance on 157, and total notifications hit an all-time high.
On cost, the honest version: for a corpus in the low thousands of documents, inference and vector storage are usually a few hundred dollars a month, and sometimes less. The real spend is the work around it. Getting a clean, versioned document set out of SharePoint, building the evaluation questions with your subject matter experts, and running the pilot will dominate the budget. Expect a first build to be a matter of weeks rather than months, and expect the document cleanup to take longer than the software.
The limitation worth naming up front: this system is only as current as your policy library. If half your documents are past their review date, an assistant will surface that problem to everyone at once. Several clients have found that useful. A few have found it uncomfortable.
PicNet builds production AI systems for Australian organisations. Talk to us about what a first project could look like.
Originally published at picnet.com.au.
Top comments (0)