DEV Community: Guido Tapia

A policy and procedure chat assistant for health organisations

Guido Tapia — Wed, 05 Aug 2026 04:45:38 +0000

Most health organisations have somewhere between a few hundred and a few thousand internal documents that tell staff how to do things. Consent forms. Incident escalation. Rostering rules. Infection control procedures. Which form goes to which committee, and by when. The documents exist, they are mostly current, and almost nobody can find the right one in under five minutes. So people ring the ward clerk, or the quality manager, or whoever has been there longest, and that person answers the same twelve questions every week.

This is the least glamorous AI use case in health and probably the best first one. It sits entirely on the administrative side, the source material is text you already own and control, and when it goes wrong the failure is visible rather than silent. This post is part of our Practical AI in Health series, and it covers what we actually build for this: retrieval-augmented generation over an internal document set, with citations, hard refusals, and an evaluation you run before anyone outside the project team touches it.

Scope it narrowly, on purpose

The first design decision is what the assistant will not answer, and it is a compliance decision rather than a product one.

The TGA's guidance on clinical decision support software sets a three-part exemption test in Schedule 4 Part 2 of the Therapeutic Goods (Medical Devices) Regulations 2002, in force since 25 February 2021. Software has to only support recommendations to a health professional, must not process medical images or signals from other devices, and must not replace clinical judgment. All three have to be met, and the TGA is explicit that calling something "decision support" does not make it exempt. Even where the exemption applies, sponsors must notify the TGA within 30 working days of supply, meet the Essential Principles, and report adverse events.

A chatbot that answers "what is our procedure for documenting a medication error" is an administrative tool. The same chatbot, if it starts answering "what dose should I give," has moved into a different regulatory conversation. So the refusal behaviour is not politeness. It is the control that keeps the system inside the scope you scoped it for. Where an answer touches anything clinically adjacent, the design requirement is that a named human signs off before the answer is acted on, and the interface says so at the point of use rather than in a footer.

On the broader regulatory picture, the Commonwealth's Safe and Responsible AI in Health Care review (final report, 2025, drawing on 69 written submissions) concluded that existing health portfolio legislation can largely accommodate AI with minor and technical amendments. There is no new AI act to comply with. Your obligations are the ones you already have around privacy, records and clinical governance.

The architecture

The shape of the system is unremarkable, which is the point:

Ingestion. Pull documents from wherever they live: SharePoint, the intranet, a policy management system. Keep the document ID, version, owner, approval date and review date as metadata on every chunk.
Chunking. Split on document structure, not fixed token counts. Policy documents have numbered clauses and headings, and those boundaries are what a person will want to be pointed at.
Embedding and index. A vector store with metadata filters, so you can restrict retrieval to current versions and to the document sets a given role is allowed to see.
Retrieval. Hybrid search (dense vectors plus keyword) usually beats either alone on policy text, because staff use exact form numbers and internal acronyms.
Generation. A prompt that instructs the model to answer only from retrieved chunks, to quote the clause, and to say it does not know when the retrieved text does not contain the answer.
Citation rendering. Every claim in the answer links to a document, a version and a section, so the reader can open the source in one click.
Logging. Question, retrieved chunk IDs, answer, citations, user feedback. This is your audit trail and your evaluation data.

A chunk record looks roughly like this:

{
  "chunk_id": "POL-CG-014-v3.2#4.1.2",
  "doc_id": "POL-CG-014",
  "title": "Open Disclosure Policy",
  "version": "3.2",
  "approved": "2025-11-04",
  "review_due": "2027-11-04",
  "section": "4.1.2 Notifying the patient",
  "audience": ["clinical", "admin"],
  "text": "..."
}

Carrying review_due through to the answer matters more than it looks. An assistant that confidently quotes a policy which expired eight months ago is worse than a search box, because it has removed the moment where the reader would have noticed the date on the cover page.

Why RAG rather than fine-tuning

We get asked this on nearly every engagement, usually phrased as "can we just train the model on our policies."

Ovadia et al. tested exactly that comparison at EMNLP 2024, putting a base model, unsupervised fine-tuning, RAG, and fine-tuning plus RAG against knowledge-intensive tasks. RAG consistently outperformed fine-tuning for both existing and entirely new knowledge, and combining the two did not reliably beat RAG on its own.

The practical arguments are stronger than the benchmark one. Fine-tuning bakes knowledge into weights, so it cannot tell you which document an answer came from, and you cannot cite what you cannot locate. When a policy is revised, RAG needs a re-index and fine-tuning needs a retraining cycle. And when a policy is withdrawn, RAG deletes the chunk while fine-tuning leaves the old text somewhere in the weights with no reliable way to remove it.

Retrieval quality is where accuracy is won. A published RAG chatbot over hospital EMR manuals, the closest analogue to an internal policy assistant we have seen in the literature, built a 5,931 question-document evaluation set and raised top-k retrieval accuracy to 97.6% by fine-tuning the embedding model rather than changing the LLM.

Citations are the safety mechanism, and they can also lie

"We use RAG" is not a safety claim. Stanford RegLab's preregistered study of commercial legal research tools, published in the Journal of Empirical Legal Studies in 2025, hand-scored 202 queries and found hallucination rates of roughly 17% for Lexis+ AI and 33% for Westlaw AI-Assisted Research, against 43% for GPT-4. RAG helped. It did not eliminate the problem, despite vendors advertising "hallucination-free" citations.

The finding that should change your design is the one about citation hallucination: an answer that cites a real but wrong source, which the authors argue may be even more pernicious than outright invention, because the presence of a reference is what makes reviewers stop checking. That is the same trap the Australian Commission on Safety and Quality in Health Care describes as automation bias, with errors of commission (acting on an incorrect recommendation) and errors of omission (failing to act when the tool misses something).

So we score citation correctness separately from answer correctness, and we show the quoted clause text inline rather than a bare reference number. If the reader can see the sentence the answer came from, checking costs three seconds instead of three minutes.

Evaluation before rollout

Do not launch on vibes and a demo to the executive team. The evaluation design we use follows a published 2025 RAG deployment: build a test set of around 100 questions with ground-truth answers reviewed by two subject matter experts, then score with RAGAS metrics that separate retrieval failure (context precision, context recall) from generation failure (faithfulness, answer relevancy).

The separation is what makes the numbers actionable. Low faithfulness with good retrieval means the model is inventing and you fix the prompt or the model. Low context recall means the policy was never retrieved and you fix chunking, embeddings or the index. Add two categories your test set will not otherwise cover: out-of-scope questions that must be refused, and questions whose answer sits in a superseded document, where the correct behaviour is to cite the current version.

The Commission's guide is structured around "before you use", "while you use" and "after you use", which maps neatly onto a rollout plan: evaluation gate, supervised pilot with feedback capture, then periodic re-evaluation as documents change.

Privacy, hosting and cost

The OAIC's October 2024 guidance treats any organisation using a commercial AI product, including purely internally, as a "deployer" with obligations under the Privacy Act 1988 and the 13 Australian Privacy Principles. Policy documents are usually not personal information, but the query logs can be, because "what do I do if I made this error" is a question about a person.

That argues for keeping the index and the logs in Australian-hosted, access-controlled infrastructure, with data-retention terms in the model provider contract that prohibit training on your traffic. The risk context is not abstract: health service providers were the most-breached sector in Australia in calendar 2025, with 225 of 1,205 notifications, ahead of finance on 157, and total notifications hit an all-time high.

On cost, the honest version: for a corpus in the low thousands of documents, inference and vector storage are usually a few hundred dollars a month, and sometimes less. The real spend is the work around it. Getting a clean, versioned document set out of SharePoint, building the evaluation questions with your subject matter experts, and running the pilot will dominate the budget. Expect a first build to be a matter of weeks rather than months, and expect the document cleanup to take longer than the software.

The limitation worth naming up front: this system is only as current as your policy library. If half your documents are past their review date, an assistant will surface that problem to everyone at once. Several clients have found that useful. A few have found it uncomfortable.

PicNet builds production AI systems for Australian organisations. Talk to us about what a first project could look like.

Originally published at picnet.com.au.

Where deterministic matching ends: cleaning and matching patient records with AI

Guido Tapia — Wed, 05 Aug 2026 04:45:34 +0000

Every organisation that runs more than one system eventually holds the same person several times over. A referral arrives with a shortened first name and no date of birth. The patient administration system has them at their old address. The billing system has a typo from a phone booking in 2019. Someone in reception creates a new record because search returned nothing useful, and now there are four.

This post is part of the Practical AI in Health series, and it covers the least glamorous work in the whole series: getting records to agree on who a person is before you build anything on top of them.

Matching is an accreditation obligation, not just data hygiene

NSQHS Standard 6 requires health service organisations to use at least three approved patient identifiers at registration, at every point of care, and on handover, transfer and discharge documents. Room number, bed number and diagnosis are explicitly not approved identifiers. Action 6.6 goes further and requires organisations to document the processes used to match patients to their intended care (ACSQHC).

That last point changes the engineering. Your matching design is an artefact an assessor can ask to see. If the answer is "the vendor's algorithm does it", you have a documentation problem as well as a data problem.

Start with the identifiers that actually hold

The Individual Healthcare Identifier is a 16-digit number assigned automatically to everyone enrolled in Medicare or with DVA, and it stays with the person for life. A Medicare number does not. A person can hold several over a lifetime, which is exactly why so many systems that keyed on it have duplicates (Australian Digital Health Agency).

The IHI is not a free pass either. The ADHA's National Healthcare Identifiers Roadmap 2023-2028 names the defects it is working through, including retired IHI records turning up in search results, failed message searches, identifier status flags that do not do their job, and poor matching outcomes for Aboriginal and Torres Strait Islander peoples. Design as though a returned identifier can be stale.

The deterministic layer does most of the work

Most duplicate pairs are boring and resolvable by rules. Normalise first, then block, then compare. Something like this, running inside the integration layer rather than in any one source system:

// 1. Trusted identifier match: auto-resolve, no review needed.
if (HasValue(a.Ihi) && a.Ihi == b.Ihi && a.IhiStatus == Active) return Verdict.Merge;

// 2. Normalised composite: strip punctuation, fold case, standardise
//    address via a locality/postcode lookup, keep nickname expansions.
var ka = Key(a); var kb = Key(b);           // (soundex(family), dob, postcode)
if (ka == kb && SimilarGiven(a, b) > 0.9) return Verdict.Merge;

// 3. Everything scoring between the thresholds goes to the grey-zone queue.
var score = Weighted(a, b);
return score >= AutoThreshold ? Verdict.Merge
     : score <= RejectThreshold ? Verdict.Distinct
     : Verdict.Review;

Tune the thresholds so the deterministic layer never merges on a guess. That leaves a queue, and the queue is where the interesting work sits.

What the grey zone looks like

A published comparison of real matching approaches gives useful numbers. Probabilistic matching returned sensitivity of 0.6366 with positive predictive value of 0.9995. Referential matching returned 0.9351 sensitivity at 0.9996 PPV (Grannis et al., JAMIA 2022). Read those two columns together. Tuned matching almost never merges two different people. What it does is miss matches, and a third of them in the probabilistic case.

Missed matches are duplicates. Duplicates mean a clinician sees part of a history, a recall letter goes to one record and not the other, and your reporting counts one person as two.

Where an LLM earns its place

The candidate pairs that survive the deterministic layer are usually hard for a reason a human can articulate. Transposed given and family names in a culturally diverse patient base. A married name in one system and a maiden name in another with the same date of birth. Two siblings at one address with dates of birth six days apart in different years. A mononymous person whose single name the source system split across two fields, which the HL7 identity matching guidance says should be submitted in the last-name field (HL7 Interoperable Digital Identity and Patient Matching IG).

An LLM is good at exactly this narrow adjudication: read two records, weigh the evidence, and explain the reasoning in a sentence a reviewer can accept or reject. Current research is worth reading before you build it, because the framing of the task changes the results as much as the choice of model. Pairwise match, comparison and selection formulations all perform differently (Wang et al., COLING 2025).

Two design requirements we treat as non-negotiable. The model never writes a merge. It ranks the queue and drafts the justification, and a named human approves each merge, with the decision and the reasoning stored against the golden record. And every merge is reversible, because you will be unmerging some of them.

Field completeness beats algorithm choice

The AIHW's COVID-19 linked data set achieved over 90 per cent linkage in every jurisdiction, with Tasmania highest at 99 per cent and Victoria carrying the largest volume at 2,536,790 individuals. The Northern Territory lagged, and the stated reason was limited address information (AIHW). No model fixes an empty field.

The same analysis found people aged 70 and over had the highest proportion of unlinked records at 9.9 per cent, and linkage fell where sex was recorded inconsistently or as 'Other'. Matching failure lands unevenly on older patients and on people whose demographics do not fit a two-value field. Report your match rates by cohort, not just in aggregate, or you will never see it.

The integration architecture

We build this as a layer beside the source systems, not inside them, using Centazio, our MIT-licensed C# integration and master data platform.

Sources publish changes into Centazio, which handles fault tolerance and near real time delivery into a central database.
Records land in a canonical Patient shape. AU Core v1.0.0, built on FHIR R4 by the CSIRO-led Sparked accelerator, is the sensible target. Note its limits: Patient carries historical names and addresses but only a single birthDate.
Deterministic rules run on write, producing merges, rejections and a review queue.
The LLM adjudicator scores and explains the queue. Humans decide.
The golden record keeps every source identifier as a cross reference, so nothing is destroyed and every merge can be walked back.
Serve matches out via the FHIR Patient/$match operation, which returns a scored bundle with a match grade. The base specification deliberately mandates no algorithm and no minimum input data set, so the accuracy is entirely yours to own and document.

Centazio came out of this kind of work. At the Cruising Yacht Club of Australia an initial cross-system reporting dashboard turned into a five-year transformation plan with the systems integrated through Centazio, and it runs in similar integration roles for clients including Guide Dogs NSW/ACT.

Costs, limits and the compliance clock

Token cost is not the expensive part, because the LLM only sees the grey zone and the grey zone is small. Reviewer time is the real budget line, so size the queue before you promise anyone a timeline. Expect a heavy first pass on historical data and a much lighter steady state.

There is also a date to plan for. The Privacy and Other Legislation Amendment Act 2024 inserts APP 1.7, requiring privacy policies to disclose automated decision making that affects people's rights or interests, commencing 10 December 2026, with infringement notices of up to $66,000 per contravention (MinterEllison). An AI-assisted decision about whether two records are the same person sits inside that.

Leaving the duplicates alone is not the safe option either. Health service providers were the most notified sector in the OAIC's 2025 breach statistics with 225 notifications, 19 per cent of a record 1,205 total, and human error drove 37 per cent of breaches in the January to June period (OAIC). Fragmented records mean more places to send the wrong document to the wrong person.

Start by measuring. Run your deterministic rules over a copy of production, count the auto-merges, count the grey zone, and have two experienced staff review a sample of 200 pairs. That gives you a baseline, a queue size and a training set, and it takes a couple of weeks rather than a couple of quarters.

PicNet builds production AI systems for Australian organisations. Talk to us about what a first project could look like.

Originally published at picnet.com.au.

From meeting audio to structured minutes in health settings

Guido Tapia — Wed, 05 Aug 2026 04:39:27 +0000

Most health services we talk to are drowning in meetings before they get anywhere near a patient. Clinical governance committees, quality and safety, credentialling, infection control, drug and therapeutics, plus multidisciplinary case conferences that can run through twenty patients in ninety minutes. Someone in the room is taking notes on a laptop while also trying to participate. The minutes land three weeks later, if they land at all, and the action items nobody wrote down quietly disappear.

This is one of the better places to put AI in a health organisation, mostly because the failure modes are containable. It is also a place where teams get into trouble quickly if they treat it as a software purchase rather than a records and consent problem. This post is part of our Practical AI in Health series and covers the version we would actually build: transcription plus structured summaries for committees and case conferences, with the clinical record left well alone.

Scope: minutes, not notes

Draw the line early and write it into the design. The system produces meeting records: attendance, agenda items, decisions, actions and owners. It does not write clinical notes, does not go near a patient's medical record, and does not generate anything that reads as advice.

That line has a regulatory meaning, not just a philosophical one. The TGA's position is that a digital scribe intended only to transcribe and translate conversations into written records is not a medical device, but one that analyses or interprets a clinical conversation, for example by generating a diagnosis or a treatment recommendation the clinician did not state, is a medical device and must be in the ARTG before it is supplied. The Safety and Quality Commission makes the same point in its ambient scribe safety scenario, noting that a tool suggesting a diagnosis can trigger ARTG requirements even when that was never the developer's intent.

For minuting, the practical consequence is a hard prompt and schema constraint. The model records what the chair declared, in the chair's words. If the meeting decided to refer a patient to radiation oncology, the minute says that. It never adds the reasoning nobody voiced, and it never fills a blank decision field with something plausible.

The pipeline

The architecture is not exotic, which is the point.

Capture: a room device or a Teams recording, one file per meeting, uploaded to a storage account in an Australian region.
Diarisation and transcription: speaker separation first, then automatic speech recognition. Output is a timestamped transcript with speaker labels.
Speaker mapping: the chair or secretary maps "Speaker 3" to a real name once, at the start of the review. Voice fingerprints stored across meetings sound efficient and create a biometric data problem you do not want.
Structured extraction: a language model fills a fixed schema, not free prose. Anything it cannot find stays null.
Review queue: a named reviewer edits and approves before anything is circulated.
Publication and retention: approved minutes go to the document management system, the raw audio goes to a deletion schedule.

The schema is what makes the output useful rather than merely readable:

{
  "meeting": "Cancer MDT - Colorectal",
  "date": "2026-07-14",
  "chair": "Dr A Nguyen",
  "attendees": ["..."],
  "apologies": ["..."],
  "quorum_met": true,
  "items": [{
    "case_reference": "MRN-XXXXXX",
    "discussion_summary": "...",
    "decision": "...",
    "actions": [{ "action": "...", "owner": "...", "due": "2026-07-28" }],
    "communication_to_gp": "pending",
    "confidence": "low|medium|high",
    "review_status": "draft"
  }]
}

That structure maps onto what Cancer Australia expects multidisciplinary teams to record: attendance, meeting frequency and the decisions made, along with a defined process for communicating outcomes and recommendations to the patient and their GP. Once the data is structured you get accreditation evidence and overdue action reports for free, which is usually the part that sells the project internally.

Consent is a state by state problem

There is no single national rule for recording a conversation. In NSW, section 7 of the Surveillance Devices Act 2007 makes it an offence to record a private conversation without the consent of all principal parties, and WA, SA, Tasmania and the ACT take a similar approach. Victoria, Queensland and the NT allow a participant to record but still penalise passing the recording on without consent. If you run services in more than one state, build for the strictest rule and use an announce and record script at the top of every meeting, captured in the minutes themselves.

Privacy sits on top of that. The OAIC's guidance on commercially available AI products names note taking and transcription assistants specifically, confirms the Privacy Act applies to both inputs and outputs, and tells deploying organisations to run a privacy impact assessment and check that vendor terms do not conflict with the consents they actually hold. Safer Care Victoria's ambient AI scribes advisory goes further and requires health services to update their privacy and health information management policies to match the specific product implemented. That is a policy job with a due date, not a paragraph in a business case.

Accuracy: plan against the right number

Vendors quote word error rates from clean single speaker audio. MLCommons' 2025 MLPerf inference benchmark used Whisper large-v3 with a reference word accuracy of 97.93% on LibriSpeech. A committee room with crosstalk, a speakerphone, mixed accents and someone unwrapping a muesli bar is a different problem, and 8 to 20% WER is the honest planning range.

Raw error rate is not the dangerous part. Koenecke and colleagues found that roughly 1% of Whisper transcriptions contained entirely hallucinated phrases that did not exist in the audio at all, and 38% of those included explicit harms such as invented associations or implied authority. Summarisation adds its own layer. In the Gold Coast Hospital and Health Service trial of ambient scribe technology across 100+ clinicians and 21 specialties, an average of only 58% of outputs were accepted without modification, with some evidence of hallucination and incorrect outputs. Four in ten needed a human to change something.

Tell your committee chairs that number before go live. A tool that saves 60% of the typing and needs 100% of the reading is still worth having, but only if people expect to read.

The review step is the product

Everything above is plumbing. The control that matters is a named person who reviews, edits and approves each set of minutes before they are circulated, which mirrors the accountability model the RACGP set out for AI scribes in general practice: explicit consent, a policy before adoption, vendor assurances on encryption, storage and destruction, and a human sign off on every output.

Build it so the reviewer cannot skip. Minutes stay in draft until approved and cannot be exported. The reviewer sees the transcript segment beside each extracted decision, with a timestamp so they can jump to the audio. Low confidence fields are flagged for attention. Approved documents carry a label recording that AI was used in their creation, which the Commission asks for and which matters if a set of minutes is ever produced in a coronial or complaints process. Case conference minutes in particular carry clinical weight even though the tool is administrative, so the chair signs them, not the secretary and not the software.

Retention, cost and the boring parts

Retention is not one number. In NSW, committee records sit under GA28, with retention varying by committee type, formal board and sub-committee records treated differently, and the secretary holding the master set while other members hold duplicates only. Public health organisations also need to check GDA21 and their patient records authority. Audio is the easier decision: treat it as working material, delete it on a short fixed schedule after approval, and say so in the consent script.

On cost, transcription compute is the small line. Reviewer time and the policy work are the real budget, along with the contingency plan the AIDH information sheet asks for, covering what the committee does on the day the tool is unavailable. AIDH also advises choosing healthcare specific tools trained on Australian data rather than generic products, released in a period when SA Health had banned open source AI scribes in its public hospitals over privacy concerns.

Start with one committee that already has a template and a secretary who is sick of typing. Run it in parallel with the current process for six meetings, measure how much editing each set of minutes takes, and only then decide whether case conferences are next.

PicNet builds production AI systems for Australian organisations. Talk to us about what a first project could look like.

Originally published at picnet.com.au.

Mining patient feedback and complaints for themes

Guido Tapia — Wed, 05 Aug 2026 04:39:23 +0000

Most health services in Australia are sitting on several years of patient comments that nobody has read from start to finish. The Likert scores get tabulated and put in the quarterly pack. The free text gets skimmed by whoever is assembling that pack, and the rest is archived. This post is part of our Practical AI in Health series, and if you are looking for a first AI project that is cheap, visible to your board and nowhere near a clinical decision, this is the one we usually point people to.

The data is already sitting there

If you run an Australian hospital you are probably already collecting AHPEQS, the ACSQHC's 12-item patient experience question set, which includes a free-text comment field. One implementation study covered 86,180 surveys across 36 private hospitals over 18 months. Nothing new needs to be collected for a theme-mining project. The corpus exists.

The volumes are worth knowing before you scope anything. In a provincial health system's inpatient survey analysis, 43.4% of adult patients and 46.9% of paediatric caregivers left a free-text comment, and topic modelling on those comments produced 86 adult and 35 paediatric topics, including elements of care that no existing survey question asked about. That last part is the argument for doing this at all. Closed items can only measure what someone thought to ask.

A single hospital's 2023 feedback corpus held 126,134 entries, 92,578 of them with free text, totalling about 1.57 million tokens at an average of 16.9 words per comment. Short comments, high volume. Trivial for a machine to process, impractical for a person to read.

Complaints are a separate stream and a more sensitive one. Ahpra received 13,327 notifications about practitioners in 2024/25, up 19% on the year before, with clinical care at 35% of all notifications and communication the second-largest issue at 4,313. Communication problems are exactly the kind of signal that shows up in free text months before it shows up anywhere formal.

Why this is the low-risk starting point

The Commonwealth's Safe and Responsible AI in Health Care legislation and regulation review found that consultees considered the current regulatory system not fit for purpose, and canvassed mandatory guardrails for high-risk settings. Analysing what patients wrote about waiting times and car parking is not a high-risk setting. There is no diagnosis, no triage, no treatment recommendation, and no patient-facing output.

It also lands on an accreditation hook you already have to satisfy. NSQHS Action 1.13 requires you to seek patient and carer feedback regularly and use it to improve safety and quality, and companion Action 1.14 requires complaints analysis to be reported to the governing body, to drive improvement, and to have identified risks recorded in the risk management system. A theme-mining project produces evidence for all of that as a by-product.

Start with a taxonomy, not a topic model

The tempting approach is to run unsupervised clustering and see what falls out. Do that second. Start with the Healthcare Complaints Analysis Tool (Gillespie and Reader, BMJ Qual Saf 2016), which was built from a systematic review of 59 studies covering 88,069 complaints and distils 729 codes into three domains and seven problem categories: quality, safety, environment, institutional processes, listening, communication, and respect and patient rights. Each is graded across four severity levels, plus stage of care and harm.

Two practical consequences. First, raters found an average of 1.94 problems per complaint letter, so your classifier has to be multi-label. Single-label classification will quietly lose half the signal. Second, a published taxonomy means your numbers can be compared against other sites and across years, which an invented cluster set cannot.

The closest published analogue to what you would be building is a 2025 cross-sectional study where 1,816 anonymous complaints from seven public primary care clinics were coded by trained humans against HCAT(GP), then classified by GPT-3.5 turbo, GPT-4o mini and Claude 3.5 Sonnet. Accuracy ranged from 58.4% to 95.5% depending on the field, agreement with human coders varied a lot (Cohen from 0.114 to 0.623), and the authors concluded further fine-tuning was needed. Thematic analysis of the same corpus found long wait times (21.6%), staff attitudes (15.8%) and appointment booking issues (10.5%) accounted for nearly half of all complaints. Read that result as: good enough to prioritise reading order, not good enough to publish unreviewed.

The architecture

Ingest from the survey platform, the complaints register and the feedback inbox into a staging store. We built Centazio for this kind of multi-source integration work, but any pipeline that gives you an immutable staging copy is fine.
De-identify before anything leaves your network. Strip or pseudonymise patient names, clinician names, dates, room and ward identifiers, and keep the mapping inside your environment.
Classify each comment against the HCAT labels with structured output, multi-label, with a confidence value and the text span the model relied on.
Run unsupervised clustering as a second pass over comments the taxonomy handles poorly. That is where the categories nobody thought to ask about turn up.
Aggregate into theme counts by site, ward and quarter, with sentiment trend lines and a simple alert when a theme's share moves outside its historical band.
Route low-confidence and high-severity items to a human review queue.

A per-comment output looks something like this:

{
  "comment_id": "c-2026-04-118322",
  "domain": "management",
  "categories": ["institutional_processes", "communication"],
  "severity": "medium",
  "stage_of_care": "in_practice",
  "harm": "minimal",
  "confidence": 0.62,
  "evidence": ["waited over two hours past my appointment time"]
}

Human sign-off is a design requirement here, not a caveat. No item reaches the risk register, a named individual, or a governing body report without a quality officer confirming the classification. The model decides what gets read first. A person decides what it means.

Privacy and governance

The OAIC's guidance on privacy and the use of commercially available AI products, published in October 2024, is the document to work from. The Privacy Act 1988 and the APPs apply wherever personal information is used to train, test or run an AI system, APP 10 accuracy obligations are heightened in an AI context, and the OAIC recommends a Privacy Impact Assessment before you introduce a new system, along with alignment to the Voluntary AI Safety Standard. Re-identification through cross-matching datasets is called out as a key risk, which matters because patients routinely name a clinician, a ward and a date in a two-sentence comment.

Budget for this from day one. The NIHR co-design study demonstrated automated free-text analysis with near real-time dashboards, and then the planned NHS-wide roll-out stalled on information governance rather than technical failure. That is the most likely way this project dies here too.

Costs and honest limits

The model bill is the smallest line item. A corpus of a million or two tokens costs very little to classify at current API prices, even if you re-run it several times while tuning prompts. The real cost is human: a few hundred comments double-coded by two staff to build a validation set, which is two to three weeks of a quality officer's time, plus the PIA and vendor review.

Some limits worth stating up front. Sentiment on 17-word comments is noisy and should be read as a trend, never as a per-comment score. Feedback volumes skew toward strong opinions, so theme counts are not incidence rates for the patient population. Severity classification is where model agreement is weakest, so treat it as a sorting aid only. And if you change model versions, re-run the back history, otherwise your trend line is measuring the model rather than the hospital.

Done properly, what you end up with is a quarterly view of what patients actually said, ranked by volume and severity, traceable back to individual comments, and defensible in an accreditation conversation.

PicNet builds production AI systems for Australian organisations. Talk to us about what a first project could look like.

Originally published at picnet.com.au.

AI-assisted medical coding: suggestions your coders can defend

Guido Tapia — Wed, 05 Aug 2026 04:33:17 +0000

Clinical coding is one of the few places in a hospital where a software error has a directly calculable dollar value. Under activity based funding, the 2025-26 National Efficient Price is $7,258 per NWAU(25), a 5.9% increase and the largest since national ABF began. A tonsillectomy is 0.7421 NWAU, about $5,386. A hip replacement at minor complexity is 4.0251 NWAU, about $29,214. Miss an additional diagnosis that shifts complexity level and you have moved five figures on a single episode.

That is why coding keeps coming up when we talk to health CIOs about where AI might actually pay for itself, and it is why we keep giving the same answer: build a suggestion engine, not a coding engine. This post is part of our Practical AI in Health series, and it covers the architecture we would put in front of a coding team, what the evidence supports, and what it costs.

The problem is scale, not competence

Separations grew about 7%, from 11.3 million in 2018 to 12.1 million in 2022, while the coder workforce ages and universities close their health information management programs. Half the previously offered undergraduate HIM courses were discontinued because enrolments were too low, which the AIHW flagged in its analysis of the coding workforce shortfall. Northern Health in Victoria moved off its legacy coding system for exactly this reason. Its health information services director Odette Taylor said the AI component reduces the need to scale up the coding workforce as separations rise, with real-time feedback to coders supporting accurate DRG and NWAU reporting.

So the demand is real. The question is what you can safely hand to a model.

Why fully automatic coding is not defensible today

The strongest single citation here is the NEJM AI benchmark by Soroush et al., which tested GPT-3.5, GPT-4, Gemini Pro and Llama2-70b on generating ICD-9-CM, ICD-10-CM and CPT codes. GPT-4 was the best of them and still managed only 33.9% exact match on ICD-10-CM. The models also produced codes that were valid-looking but non-billable, or simply fabricated. That was code lookup from a supplied description, which is easier than abstracting a code from a discharge summary.

Now add the Australian complication. Admitted episodes here are coded to ICD-10-AM/ACHI/ACS Thirteenth Edition, in force for separations from 1 July 2025, with AR-DRG V12.0 sitting on top. AR-DRG V12.0 assigns Diagnosis Complexity Levels across 11,057 in-scope ICD-10-AM codes. A general-purpose model trained on the American clinical modification does not know that classification, and the editions change on a three-year cycle. Whatever you build has to be versioned to the edition in force on the separation date, not to whatever the model absorbed during training.

The assisted configuration is a different story. A 13-week randomised controlled trial with ten certified coding specialists, published in npj Digital Medicine, found AI-assisted workflows significantly reduced coding time while maintaining accuracy. Satisfaction varied with coder experience and certification, which matters more than it sounds: a tool your senior coders distrust will be worked around.

HIMAA's national guideline for AI-generated clinically coded data lands in the same place, listing human expertise and human-in-the-loop among its seven considerations, and noting honestly that published evidence of benefit in the Australian context is still lacking. Nobody has proven this here yet. Design accordingly.

The architecture

The pattern we would build looks like this:

Documents in via HL7 v2 or FHIR from the EMR and PAS: discharge summary, operation reports, progress notes, pathology. Getting these flowing reliably and repeatably is ordinary integration work, which is what we built Centazio for.
A code index built from the licensed IHACPA code lists for each edition, held separately per edition, with the separation date deciding which index is queried.
Retrieval first, generation second. The model never emits a code string freely. It nominates candidates from the retrieved set, and anything not present in that edition's index is discarded before it reaches a human.
A rules layer applying Australian Coding Standards checks: principal diagnosis sequencing, additional diagnosis criteria, mandatory ACHI pairings.
A coder-facing panel that shows each suggested code with the exact text span it came from, so review means reading two sentences rather than re-reading the record.
An append-only audit log, written before the coder sees anything.

The audit record is the part people skip and later regret. It should look roughly like this:

{
  "episode_id": "",
  "separation_date": "2026-03-14",
  "classification_edition": "ICD-10-AM 13th",
  "grouper_version": "AR-DRG V12.0",
  "model_id": "coder-assist-2026-02",
  "index_snapshot": "sha256:",
  "suggested": [{"code": "J45.0", "confidence": 0.82, "evidence_doc": "DS-4471", "span": [1180, 1246]}],
  "coder_id": "",
  "action": "rejected",
  "final_code": null,
  "reviewed_at": "2026-03-19T09:41:12+11:00"
}

That log gives you two things. Externally, it demonstrates that a qualified human assigned every code and that the tool suggested rather than decided. Internally, it is your quality feedback loop. Coding errors are systematic, not random: external audits of 55 hospitals across two states, covering 6,300 records, found a 5.9% predicted DRG mismatch rate, and at one hospital a 5.6% mismatch still translated to nearly $8 million less funding, because one error in a high-value DRG repeated many times. A log keyed by code, coder and DRG surfaces that repetition. Spreadsheets of aggregate accuracy do not.

Regulatory and privacy position

A coder-facing suggestion tool generally sits outside TGA medical device regulation under Exclusion 14G, which covers software for the administration or management of health processes including financial records, claims and billing. That exclusion holds only if every function meets the criteria and the tool does not influence clinical decision-making. The moment someone asks for the suggestions to be surfaced to treating clinicians during the episode, you have changed the intended purpose and you need to re-examine the classification.

Governance can follow the ACSQHC's AI clinical use guide, released in August 2025 and structured around before you use, while you use, and after you use. The three phases map cleanly onto evidence review, coder training with monitoring, and scheduled re-evaluation after each classification edition change.

Privacy is the part that should reach the board. These systems ingest full discharge summaries, operation reports and progress notes. Health service providers were the most-breached sector in the OAIC's 2025 notifiable data breach statistics, at 19% of notifications (225 of 1,205). Decide early whether inference runs in Australian-hosted infrastructure, who can query the index, and how long request payloads are retained. Those answers are harder to change later.

What it costs, honestly

Budget for licensed IHACPA classification products, a re-index and re-validation every three years when the edition changes, and inference costs that scale with document volume rather than with episodes. The largest cost is not the model. It is assembling a gold-coded evaluation set from your own separations, ideally auditor-reviewed, plus the coder hours to run a proper before-and-after measurement. Without that set you cannot tell whether the tool is helping or quietly teaching your team to accept plausible wrong codes, which is the failure mode that worries us most.

Start on a single high-volume specialty, measure minutes per episode and post-audit mismatch rate against a matched baseline, and keep the coder as the person who assigns the code. That configuration is the one the evidence supports.

PicNet builds production AI systems for Australian organisations. Talk to us about what a first project could look like.

Originally published at picnet.com.au.

Predicting appointment no-shows: what actually works in an Australian clinic

Guido Tapia — Wed, 05 Aug 2026 04:33:12 +0000

Every conversation about AI in health at the moment is about language models. Meanwhile the problem that most reliably wastes clinic capacity, the patient who does not turn up, is best solved with a technique that has been sitting on the shelf since about 2015: gradient boosted trees on a table of appointment records.

We have been building these models at PicNet since well before the current wave. Old posts on this blog about XGBoost still get fetched by AI assistants, which is mildly funny, because the underlying advice has not changed much. Tabular data, careful feature engineering, honest evaluation. What has changed is that health services now have the appointment history and the reporting infrastructure to make it worth doing. This post is part of our Practical AI in Health series, and it is the one where the answer is deliberately unfashionable.

Measure your own base rate before you model anything

The single most common mistake is assuming a no-show rate from the literature. Australian did-not-attend rates vary enormously by service and location. A regional medical imaging department recorded 724 failures to attend across 13,458 appointments, a rate of 5.4%. A surgical outpatient clinic in Mount Isa recorded 18.3% across 6,267 appointments. Those two services need completely different models, different thresholds and, frankly, different business cases.

There is a related trap in the published benchmarks. A review of 52 no-show modelling papers from 2010 to 2025 found that logistic regression appears in 68% of them, that the best models land between AUC 0.75 and 0.95, and that a large number of studies re-use the same 2015-16 Brazilian dataset (Sciencedirect). If a vendor quotes you an accuracy figure without naming the population it was measured on, assume it does not transfer.

So step one is not a model. It is a query against your PAS or booking system that gives you attendance by clinic, by modality, by referral source and by month for the last two or three years. If that query is hard to write, fix that first. It is also the data you need for the business case.

Features that earn their place

The useful features are mostly logistical rather than clinical. The MRI deployment we keep coming back to below used 21 categorical and numeric fields pulled straight from the radiology information system and appointment system, with no patient medical conditions at all (AJR). Typical shape:

Lead time between booking and appointment, which is usually the strongest single signal
Prior attendance history for the patient, including counts of previous no-shows and cancellations
Number of times the appointment has been rescheduled
Appointment day, hour and session
Modality or procedure type, and the ordering department or referring practice
Appointment modality, since telephone and telehealth consultations were associated with better attendance in the Mount Isa cohort (adjusted OR 0.57 and 0.62)
A travel proxy such as distance band or Modified Monash Model category

Two notes on the last one. Distance genuinely predicts non-attendance, and it is also a measure of how hard your service is to reach. In the Mount Isa data, Indigenous status was the strongest independent predictor of non-attendance at an adjusted OR of 3.72 (95% CI 3.22 to 4.29), which the authors attribute to systemic and logistical barriers rather than anything about the patients. A model trained on those fields will happily learn that access inequity is patient risk and present it back to you as a risk score. Decide up front what you will do with that, because it determines whether the project improves equity or entrenches it.

Class imbalance: do less than you think

At a 5% base rate the instinct is to reach for SMOTE or class weights. Resist it. A real-world evaluation of imbalance correction found that cost-sensitive learning, random over- and under-sampling and SMOTE did not improve AUROC or AUPRC, degraded calibration, and inflated the proportion of patients classified as high risk by up to 62.8%. The naturally trained model had the highest net benefit in decision curve analysis across clinically relevant thresholds (medRxiv).

Calibration is what you actually need, because the output feeds a capacity decision. Report AUPRC and calibration measures such as Brier score and log loss alongside AUROC, and set the operating threshold from the cost ratio of the intervention rather than defaulting to 0.5. Something like this, with your own numbers:

# your finance team's numbers, not ours
cost_of_reminder_call = 5.00     # a few minutes of admin time
cost_of_empty_slot    = 250.00   # idle clinician and equipment

# indifference threshold, before capacity limits
threshold = cost_of_reminder_call / cost_of_empty_slot   # 0.02

In practice the threshold you deploy is set by how many calls your admin team can make in a day, not by the arithmetic. Which is fine. Rank by predicted probability, take the top N you can service, and use the cost ratio as a sanity check on whether N is too small.

What good accuracy actually looks like

A deployed outpatient MRI model built with XGBoost on 32,957 appointments, at a 17.4% no-show rate, achieved a ROC AUC of 0.746 with an optimised F1 of 0.708, precision 0.606 and recall 0.852 (AJR). That is a long way from the 0.90+ figures you see quoted in richer-data studies, and it was enough. Human attendance behaviour is noisy and a lot of the variance is unrecorded: a lift home fell through, a shift changed, the letter went to an old address.

If your first model lands around 0.72 to 0.78 AUC, you have a normal, usable model. Chasing 0.85 with more features is usually worse value than spending the same effort on what happens after the score is produced.

The model is half the system

The same MRI team used the score to rank appointments and place a phone reminder to the top 25% one business day out, run over six months. No attempt to classify every appointment as show or no-show. That framing matters because it turns a probabilistic output into a fixed daily work queue for the people who actually make the calls.

On which intervention to attach, the JAMIA rapid review of model-triggered interventions is the most useful source available. It found high-certainty evidence that model-triggered SMS reminders reduce no-shows (median RR 0.91), moderate certainty for phone calls (median RR 0.61) and patient navigators (RR 0.55), and uncertain evidence for model-driven overbooking. The same review found almost no full cost-effectiveness analyses, no formal acceptability evaluation, and only half of trials reporting race or ethnicity, so effect variation across equity-relevant groups could not be assessed.

Overbooking deserves a specific warning. Samorani and colleagues show what happens when you pair an accurate classifier with a cost-minimising scheduler: the highest-risk group is systematically assigned the slots with the longest waits, and dropping demographic fields does not fix it, because postcode, language and prior utilisation act as proxies (PMC). Their recommendation is to decouple the prediction from the scheduling objective.

There is a workable alternative. A US health system running a commercial EHR's built-in no-show predictor rejected the overbooking action as ethically problematic and used the score to trigger patient-positive responses instead: flexible appointment times, telehealth options, help with transport and childcare. Across a 12-clinic pilot they saw a 9% mean reduction in no-shows (Health Affairs). Same model, different action, different distributional consequences.

Whatever the action, a person stays in the loop. The score selects who gets contacted and offered support. It never cancels, deprioritises or reassigns an appointment on its own, and any output that touches a clinical pathway goes to a named human for sign-off. Build that in as a design requirement rather than adding it as a caveat later.

The regulatory boundary

Keep the scope administrative. Software intended for the prediction or prognosis of disease can meet the definition of a medical device under s41BD of the Therapeutic Goods Act, and the Clinical Decision Support Software exemption (in force since 25 February 2021, Schedule 4 Part 2 of the Therapeutic Goods (Medical Devices) Regulations 2002) requires all three of its criteria to be met, plus notification to the TGA within 30 working days of supply and ongoing compliance with the Essential Principles (TGA guidance). A model that predicts whether someone will attend an appointment is not predicting disease. A model that starts inferring clinical deterioration from attendance patterns is a different product, and you should get advice before you build it.

What a first project looks like

Six to ten weeks is realistic for one clinic or modality. Two weeks on data extraction and base-rate analysis, two to three weeks building and validating the model, and the remainder on the daily work queue, the calibration monitoring and the operational rules for who calls whom. The build is not expensive. The part that decides whether it works is the intervention design and the willingness to look at how the benefit is distributed across patient groups after six months.

PicNet builds production AI systems for Australian organisations. Talk to us about what a first project could look like.

Originally published at picnet.com.au.

Summarising clinical and referral documents with AI

Guido Tapia — Wed, 05 Aug 2026 04:27:06 +0000

Every health service we talk to has the same pile: referrals arriving as faxed PDFs, scanned handwriting and email attachments, plus discharge summaries written at the end of a long shift. Someone has to read all of it and pull out the same dozen facts every time. It is slow, it is boring, and things get missed.

The missing part is documented. A retrospective audit of a tertiary medical oncology unit against the Australian National Guidelines found that "procedures" were absent from 82% of discharge summaries, "ceased medicine" from 38% and "information provided to the patient" from 8%. Follow-up arrangements were recorded in 91% of summaries, but exact appointment dates were often missing. That is the problem worth attacking, and it is an administrative one.

This post is part of our Practical AI in Health series. Here I want to cover how we build document summarisation that survives contact with a clinical governance committee.

Extract first, then write prose

The most common mistake is to hand the whole document to a model and ask for a summary. You get fluent paragraphs that nobody can check against anything.

The alternative is to treat the job as structured extraction. Break the document into segments, pull named entities and relations out of each one, map the results to a schema you already use, and only then generate readable text from the structured record. A published pipeline doing exactly this (OCR, chunking, entity and relation extraction, then mapping to FHIR R4 and OMOP) reported roughly 95% accuracy and F1 across seven medication data fields covering 5,789 extracted medications, though validation was on only 34 patients. Small cohort, but the design point stands: fields you can validate beat prose you cannot.

A working architecture looks like this:

Ingest and OCR, with the original page image retained for every extracted value.
Deterministic segmentation into sections before any model sees the text.
An extraction pass that returns a fixed schema and refuses to emit a value without a supporting span from the source.
Validation against terminology sets, date logic and internal consistency (a ceased medicine that never appears in the admission list is a flag, not a fact).
A completeness check that lists which required fields were not found, because absence is the failure mode you will hit most.
A narrative pass restricted to the validated structure, never the raw document.
A review screen where the clinician or intake officer sees each field next to the quoted source text, approves or corrects, and signs off.

Our extraction output looks something like this:

{
  "field": "ceased_medicines",
  "value": ["metoprolol 25 mg BD"],
  "evidence": [{ "doc_id": "dc-2291", "page": 2,
                 "quote": "Metoprolol ceased 14/06 due to bradycardia" }],
  "confidence": 0.61,
  "status": "needs_review"
}

No span, no field. It is a boring rule and it removes an entire class of argument later.

What actually goes wrong

Fabrication gets the headlines; omission is the more frequent problem. A vendor-neutral evaluation covering 18 experimental configurations and 12,999 clinician-annotated sentences measured a 1.47% hallucination rate and a 3.45% omission rate. The same paper's baseline is worth keeping in mind when someone demands perfection: human-written notes average about 1 error and 4 omissions each, and an earlier BART-based model produced 3.9 errors and 6.6 omissions per note. Your review process should ask "what did it drop?" at least as often as "what did it invent?"

Grounding is not a nice-to-have. When researchers built 300 doctor-designed vignettes and seeded each with a single fabricated lab value, sign or disease, models repeated or elaborated the planted error in up to 83% of cases, with an overall hallucination rate of 65.9% under a default prompt. A mitigating prompt brought that down to 44.2%. Two things follow. Prompt engineering helps and is nowhere near sufficient. And referrals contain errors already, so any pipeline that free-runs over the source text will confidently launder those errors into your record.

Confidence flags earn their place, but be honest about what they are. A model's self-reported confidence number is close to meaningless on its own. What works in practice is agreement between two independent extraction passes, whether a matching source span was found, and whether the value validates against a terminology set. Combine those, set a threshold, route everything below it to a human. Everything above it also goes to a human, just with less friction.

Structure the referral, let a clinician triage it

There is a temptation to go one step further and have the system assign urgency. Queensland Health researchers tried automated categorisation of 17,378 ENT referrals from two hospitals between 2019 and 2022 against the Clinical Prioritisation Criteria, and achieved 53.8% agreement with the actual assigned triage category. Extraction of referral content is a solved-enough engineering problem. Assigning urgency is not, and the clinician making that call needs to be a design requirement, not a disclaimer in the footer.

Keeping health information onshore

The OAIC's October 2024 guidance recommends as best practice that organisations do not enter personal information, particularly sensitive information, into commercially available AI products, and flags APP 3, 6, 8 and 10 as live obligations. Health information is sensitive information. An offshore API call is a cross-border disclosure that the health service remains accountable for. The ACSQHC's ambient scribe scenario is blunter still: the data must be stored and processed in Australia unless the patient has given explicit consent for offshore handling.

That leaves three practical options. A commercial API deployed in an Australian region with contractual guarantees on retention and training is the cheapest to run and the heaviest on procurement and privacy paperwork. Open-weight models on infrastructure you control give you residency, version stability and no vendor changing the model underneath you, in exchange for fixed GPU spend and owning your own evaluation. A hybrid puts a smaller local model on extraction and de-identification, with anything larger only ever seeing de-identified text, which adds moving parts and relies on de-identification you should not assume is perfect.

On cost, per-token spend is usually the smallest line item. For a few thousand documents a month it is rounding error next to the clinician review time and the validation work. Self-hosting shifts cost to fixed GPU hours, which pays off at volume or when residency requirements make the decision for you. Budget most of your money for evaluation, not inference.

Validation does not travel

Victoria's May 2025 sector advisory on ambient AI scribes sets a minimum implementation standard and states plainly that a solution evaluated in one context cannot be assumed to transfer between clinical settings, or from an overseas training context to Australian practice. Good accuracy on GP referrals tells you little about oncology or maternity. Budget for per-specialty validation with your own documents and your own clinicians.

Watch your roadmap too. TGA regulation is technology-agnostic and turns on intended purpose. A tool that drafts documentation is generally not a medical device, but the TGA's own worked example is a scribe that later gains a feature suggesting diagnoses or treatments not mentioned in the consultation, at which point it is regulated. The TGA published findings on strengthening medical device software regulation in July 2025 and has further consultation running through 2025 and 2026, so the boundary is being actively redrawn.

The ACSQHC guidance also asks for consent processes, summaries that meet medico-legal requirements, records labelled to show AI was involved, and regular performance review. Build the labelling and the audit log in from the first sprint. Retrofitting them is miserable.

Australian deployment evidence exists but is early. Gold Coast Hospital and Health Service ran a 16-week ambient scribe trial across outpatient specialties in 2024, evaluated through patient and staff surveys, interviews, scribe outputs and eMR review, and reported positive staff and patient experience alongside note quality and efficiency gains. Useful, and outpatient-weighted. Treat it as a reason to run your own measured pilot, not as proof the problem is solved.

Start with one document type and one clinic. Measure omissions against a clinician-reviewed sample before and after. If it does not reduce the fields that go missing, it is not worth deploying.

PicNet builds production AI systems for Australian organisations. Talk to us about what a first project could look like.

Originally published at picnet.com.au.

Building a daily AI digest of global health news

Guido Tapia — Wed, 05 Aug 2026 04:27:02 +0000

Most health IT managers I talk to have the same low-grade problem. There is a folder of newsletters they meant to read, a couple of industry sites they check when things are quiet, and a vague worry that something important happened last week and they missed it. Nobody has an hour a day to scan forty sources.

We built a small thing to fix that for ourselves. PicNet's internal monitoring platform, which already watches servers, jobs and integrations, now runs a scheduled AI news-summary task. Every morning it emails a digest of what happened in the last 24 hours across the sources we care about, each item summarised from the actual article with a link back to it. It costs a few cents a day to run. This post describes how it works, what we got wrong first, and how you would point the same pattern at health news. It is part of our Practical AI in Health series.

Why bother automating a reading habit

Two reasons, one boring and one less so.

The boring one is that the sector genuinely does move faster than a monthly catch-up. Health service providers were the most-breached Australian sector in 2025, with 225 of 1,205 notifications, ahead of finance and the Australian Government, in the highest annual total since the scheme began in 2018. In the January to June 2025 half, health again led the sector table and the average cyber incident affected just over 10,000 people. If a vendor you share data with turns up in the news on a Tuesday, you want to know on Tuesday, not when the six-monthly statistics come out.

The less boring reason is that automated news triage is established public health practice, not a novelty. WHO launched version 2.0 of its Epidemic Intelligence from Open Sources system in October 2025, a platform now used by more than 110 Member States that categorises open-source articles daily using NLP and machine learning. A 2025 evaluation across WHO's 47 African region countries found signals for more than 80% of notified public health events in 28 countries, and in 22 countries at least half of events were picked up before official notification. Our digest is a very small version of the same idea, aimed at an IT manager's inbox rather than an outbreak response team.

The pipeline

The whole thing is five steps on a schedule. There is no vector database, no agent framework, no fine tuning.

A scheduled task fires each morning and pulls the last 24 hours of items from a list of RSS and Atom feeds.
Titles and feed summaries go to an LLM with a short brief describing our audience. It returns the subset worth expanding, usually 10 to 20 items out of 60 or 80.
The selected URLs get scraped and reduced to article text.
Each article is summarised individually, in a separate call, from the scraped text only.
The summaries are assembled into an HTML email with source links and sent.

Splitting selection from summarisation matters more than it looks. Feed metadata is cheap and often enough to judge relevance, so you only pay to scrape and read the articles that survive the first pass. It also means the expensive step never sees the noise.

Grounding, which is the whole trick

The single design decision that makes this trustworthy is that the model never answers from memory. It summarises supplied text and nothing else.

There is good evidence for why. An EBU study led by the BBC, covering 22 public service media organisations across 18 countries and more than 3,000 evaluated responses, found 45% of AI assistant answers about news had at least one significant issue. Sourcing was the worst category at 31%, with 20% showing major accuracy problems including hallucinated details and outdated information. Those are failures of ungrounded recall. Ask a chatbot "what happened in health news today" and you are sampling exactly the behaviour the study measured. Hand it an article and ask it to compress that article, and most of that failure mode disappears.

The prompt is short and mostly prohibitions:

Summarise the article below in 2 to 4 sentences for an
Australian health IT audience.

Rules:
- Use only facts stated in the supplied text.
- If a number, date or organisation is not in the text,
  do not include it.
- Do not add background, context or implications.
- If the text is a paywall stub, navigation or an error
  page, reply exactly: SKIP.

ARTICLE:
{{text}}

The SKIP path earns its keep. Scrapers fail quietly, and without an escape hatch the model will cheerfully write a confident summary of a cookie banner. We drop anything that returns SKIP and log it, and a rising SKIP rate is usually the first sign a source has changed its markup.

Every item in the email carries its source link. If a summary reads oddly, the reader is one click from the original. That is the human check, and it should stay in the design even when the summaries get good.

The other lesson from that study is to re-test occasionally. Model behaviour is not fixed: Gemini had significant issues in 76% of responses in that round, while the overall rate had improved five percentage points on the BBC's February 2025 replication. Whichever model you pick, sample a week of its output against the source articles once or twice a year.

The scrape step

Boilerplate removal is the part that quietly ruins output quality. Raw-text extractors like BeautifulSoup get near-perfect recall of about 0.99 but precision around 0.50, according to Trafilatura's published evaluation, which means half of what you send the model is navigation, related-article teasers and footer junk. That is not just wasted tokens. It gives the model competing headlines to blend into the summary.

The same evaluation reports Trafilatura as the strongest single tool by ROUGE-LSum mean F1 in Bevendorff et al.'s SIGIR 2023 comparison, and the most efficient open-source library in ScrapingHub's benchmark. We use a dedicated extractor for that reason. Paywalled and JS-rendered sources will still fail; the honest answer is that you accept partial coverage rather than building a scraping arms race.

Cost

This is the part that surprises people. GPT-5 Mini lists at roughly US$0.25 per million input tokens and US$2.00 per million output tokens, with Azure pricing as low as US$0.125 per million input. Scraping and summarising 50 articles at around 1,500 tokens each is about 75K input tokens, so roughly US$0.02 a run before any prompt-caching or batch discount. Call it a dollar a month in tokens.

The real cost is the fifteen or twenty hours of engineering to build it and the ongoing attention when a feed dies. That is still cheap compared with an analyst reading feeds each morning, and much cheaper than a bespoke media monitoring subscription.

Governance, briefly

A digest built entirely on public news is about the safest first AI project a health organisation can run. No patient data goes near the model, which sidesteps the OAIC's clear line that organisations should not enter personal or sensitive information into publicly available genAI tools. The same guidance still applies: APP 10 requires reasonable steps to ensure accuracy given genAI's probabilistic nature, and a Privacy Impact Assessment before introducing an AI system is recommended.

Nothing here is clinical, and it should stay that way. This tool summarises administrative and industry news. It does not interpret patient data and it makes no recommendation anyone acts on without reading the source.

The Australian Commission on Safety and Quality in Health Care's AI Clinical Use Guide, published in August 2025, structures its advice as before you use, while you use and after you use, and names automation bias as a core risk. That structure works for an internal tool too: decide up front what the digest is for, keep source links visible while people read it, and review a sample of output periodically. Automation bias is the live risk here. Once a digest arrives every morning, people start assuming that anything not in it did not happen.

If you are in a state health service or another public agency, you will also need to map the tool against the National Framework for the Assurance of AI in Government, agreed in June 2024, whose five assurance cornerstones sit over Australia's eight AI Ethics Principles. A low-risk internal tool is a good candidate for walking that process end to end while the stakes are small.

Where to start

Pick eight to ten feeds you would read if you had time, including at least one Australian trade source. Run selection and summarisation as separate calls. Attach the source URL to every item. Add the SKIP rule before you need it. Send it to yourself for a fortnight before you send it to anyone else, and read the linked articles for a few of the items each day to see whether the summaries hold up.

PicNet builds production AI systems for Australian organisations. Talk to us about what a first project could look like.

Originally published at picnet.com.au.