Most of the questions that clog an HR inbox or an operations channel have already been answered in writing. The leave policy says what happens when a public holiday falls during annual leave. The installation manual says which bracket suits a tiled roof. The problem is that the answer sits on page 34 of a PDF someone uploaded to SharePoint in 2019, so people ask a colleague instead, and the colleague guesses.
This is the least glamorous use of a language model and the one with the clearest payback. It is also the pattern we are asked about most often, which is why it appears early in Practical AI in Business Operations. The approach below is what we build for clients outside health as well as in it: HR policies, operations manuals, product documentation, supplier agreements, safety procedures.
What the system actually does
A user asks a question in plain English. The system finds the passages in your documents most likely to contain the answer, hands those passages to a language model, and asks it to answer using only what it was given. The answer comes back with links to the source documents and the sections used. If the passages do not contain the answer, the system says so.
That last behaviour is the one that decides whether staff trust the thing. A general model will happily produce a confident paragraph about your redundancy process based on what redundancy processes usually look like. That answer is worse than no answer, because it sounds like it came from your policy.
Architecture sketch
- Ingestion. Pull documents from wherever they live (SharePoint, Confluence, a file share, a DMS) on a schedule. Convert to text, keeping headings and page numbers so a citation can point at a section rather than a 90 page file.
- Metadata. Every chunk carries the document title, owner, effective date, the version it supersedes, and which entity or state it applies to. This is where most of the value hides. Without it you cannot filter a Queensland question away from a Victorian policy, and you cannot tell the user the answer is current as at a date.
- Chunking. Split on document structure, not on a fixed character count. A clause with its heading and its parent heading retrieves far better than 800 characters that start mid sentence.
- Index. Embeddings plus keyword search, with results merged. Pure vector search misses exact terms like award classification codes or part numbers. Keyword search alone misses "can I take leave at half pay". You want both.
- Answer. The model receives the question, the top passages, and a strict instruction set. It returns structured output, not free text.
- Logging. Every question, the passages retrieved, the answer, and any user feedback. This is your evaluation data and your audit trail.
We usually put the index in Postgres with pgvector rather than adding a new vendor. For corpora in the tens of thousands of chunks, which covers most SMEs, it is fast enough and it keeps your document content inside infrastructure you already govern under the Privacy Act.
The prompt contract
The instruction to the model is short and it does not negotiate:
Answer using only the numbered passages below.
Every sentence in the answer must cite at least one passage id.
If the passages do not contain enough to answer, return not_in_corpus.
Do not use general knowledge about how policies usually work.
And the output is a schema, so the application can check it before a human ever sees it:
{
"status": "answered",
"answer": "Annual leave accrues progressively...",
"citations": [
{"passage_id": 3, "document": "Leave Policy v4.2",
"section": "5.1 Annual leave", "effective": "2025-07-01",
"url": "https://.../leave-policy-v4-2.pdf#page=12"}
]
}
Then verify before rendering. If status is answered and citations is empty, drop the answer and show the retrieved passages instead. If a cited passage id was not in the set you supplied, treat it as a failure. These checks cost nothing and they catch the most embarrassing errors.
Make the citation a link to the source document at the right page, not a file name. Staff click through more often than you would expect, and the ones who click are the ones who will tell you when a policy is out of date.
Evaluation before anyone else sees it
Do not roll this out on a demo. Build a question set first.
Take 100 to 200 real questions from your HR inbox, service desk tickets or the operations channel. For each one, have the person who currently answers it write down the correct answer and the document and section it comes from. That is a day of someone's time and it is the highest value day in the project.
Then run the set and grade three things: whether the answer is correct, whether the citation points at the right section, and whether the system refused when it should have answered. Add a second set of 30 or so questions you know are not covered by the corpus, including a few that sound like they should be. Count how often the system answers those anyway. That false answer rate is the number to watch, and it is the one that moves when someone changes a prompt, swaps a model version or adds a new document set.
Re-run both sets on every change. It takes minutes once it is scripted, and it turns "the new model seems better" into something you can check.
If your documents are clinically adjacent, keep the assistant on administrative ground: rosters, credentialling, procurement, incident reporting workflow. Anything that could shape care needs a named clinician signing off the answer before it reaches anyone, designed in as a workflow step with an approval record, not added later as a disclaimer.
What it costs to run
Three lines, in descending order of size.
The first is corpus preparation. Someone has to decide which documents are authoritative, retire the superseded ones, and attach the metadata. In most engagements this is the bulk of the effort and it is internal staff time, not licence fees. It is also work that pays off whether or not you build the assistant.
The second is ongoing ownership. A document set that nobody maintains degrades, and a confidently cited answer from a withdrawn policy is a real problem. Budget a few hours a month for someone to review flagged answers and update the corpus.
The third is inference, and it is usually the smallest. Work it out yourself rather than trusting anyone's estimate: multiply the tokens you send per question (retrieved passages plus the question plus the answer, typically a few thousand) by your expected monthly question volume, then check your provider's current published price. A department sized deployment of a few thousand questions a month generally lands in the range of a single software subscription. Embedding the corpus is a one off charge plus a trickle for new documents. If you self host the index, add nothing for the database because it is already there.
When a wiki search is honestly good enough
Sometimes it is. If your corpus is a well structured intranet with good headings, your staff know the vocabulary, and volume is low, decent search plus a table of contents will solve the problem for the cost of nobody's afternoon. The same goes for a corpus small enough that a person can hold it in their head.
The pattern earns its keep when the documents are long, badly structured or scattered across systems, when the answer requires reading across several documents, when users do not know the internal terminology, or when the same questions arrive dozens of times a week. It does not help at all when your documents contradict each other. Fix that first, because retrieval will find both versions and the model will pick one.
PicNet builds production AI systems for Australian organisations. Talk to us about what a first project could look like.
Originally published at picnet.com.au.
Top comments (0)