An auditor sits across from you with a single page of questions. They are not interested in your model architecture, your prompt engineering, or your evaluation harness. They want to know one thing: when your AI agent answered a clinician's question last Tuesday, what data did it see, who authorized that access, and can you prove it.
This is the moment most clinical AI systems quietly fail. Not because the team did not care about compliance — they did — but because the system was architected to make AI work, not to make audits work. Authorization was an application-layer concern. Audit logs captured user clicks but not model retrievals. The vector database lived outside the compliance perimeter. The agent reached data through generated queries that were never persisted in a form an auditor could reconstruct.
Clinical AI is shipping into hospitals now. The first wave of HIPAA audits and security reviews of these systems is already underway. The architectural patterns most teams are using were not designed for regulated workloads, and they do not hold up under serious scrutiny. This article is the question list I wish more teams had on the wall before their first audit.
What a HIPAA Audit Is Actually Looking For
HIPAA audits, whether driven by the Office for Civil Rights or by a covered entity's own internal review, do not test whether your AI is good. They test whether your handling of Protected Health Information is defensible. The Privacy Rule, the Security Rule, and the Breach Notification Rule define the structure. The questions an auditor asks fall into a narrow set of categories that map to those rules — and they are the same questions, in roughly the same order, every time.
There are six categories worth designing for explicitly. Each is a question you should be able to answer in minutes, not weeks, with evidence drawn directly from your system.
1. Who saw what, and when?
This is the foundational audit question. For any patient, any record, any field, the auditor expects you to produce a record of every access — read or write, by a human or a system — with a timestamp, an actor, and a reason. The HIPAA Security Rule requires audit controls; the Privacy Rule's accounting of disclosures provision adds a patient-facing layer that requires the same data, in a different format.
In a non-AI system, this is hard but tractable. Application-level access logs, database audit triggers, and a periodic export are usually enough. In an AI system, this question fragments. A clinician asks an agent a question. The agent retrieves five structured records and three free-text notes. It calls a model. The model returns a draft. The clinician sees the draft. Which of those five structured records counts as a disclosure to the clinician? All of them, even the ones that did not influence the answer? The ones that were quoted in the response? The ones the clinician scrolled past in the source citations? The auditor will ask, and "the model decided what to surface" is not an answer that survives the meeting.
What the architecture must support: every retrieval the agent performs — structured query, vector search, tool call — must produce an audit record tied to the requesting user, the clinical justification, the records returned, and the records ultimately surfaced. The records returned and the records surfaced are different sets, and both matter.
2. Was the access authorized?
Audit logs are necessary but not sufficient. The next question is whether each access was permitted under the user's role, the patient's consent directives, the purpose of use declared at session start, and the minimum-necessary standard. If a behavioral health note appears in the agent's retrieval set for a request that did not require it, the system has failed the test, even if the note never reached the user.
The hardest part is that authorization in clinical systems is contextual. The same physician has different access to the same patient depending on whether they are the patient's attending, the patient's covering provider, a consulting specialist, or none of the above. A psychiatric note may be visible to the patient's psychiatrist but not to a cardiologist consulting on the same encounter. A break-the-glass declaration permits access that would otherwise be denied, but creates an obligation to document the justification.
What the architecture must support: authorization belongs in the data layer, not the application layer. Every read — structured, vector, tool-mediated — must pass through the same policy engine that knows about role, relationship, consent, purpose of use, and minimum necessary. Filtering after retrieval is too late; the auditor will ask whether the agent saw the data, not whether it surfaced the data.
3. What did the model actually see?
This is the question that separates AI systems from the systems that came before them. When a model produces a response, the audit record must show not only the user's query and the model's output but the full prompt the model received — including the retrieval context, the system instructions, and any tool results that were inlined. If the model saw a sentence from a note in its prompt, that sentence is part of the disclosure record, whether or not it appeared in the final response.
The corollary is that any de-identification you applied to the prompt is also part of the audit. If your egress gateway redacted patient names before sending the prompt to an external model, the auditor will ask to see the redaction logs, the redaction rules, and evidence that the rules worked correctly on this specific prompt. Safe-harbor de-identification has eighteen specific identifier categories; expert-determination de-identification has a different standard. The auditor will ask which one you used and how it was validated.
What the architecture must support: every model invocation produces an audit event recording who caused it, which model received the prompt, whether de-identification was applied, and whether the prompt left the compliance boundary. The prompt and response themselves go to a separate prompt store, linked to the audit event by a single ID. The audit event records the decision; the prompt store carries the content. Both are queryable years later. "We don't log prompts because they're large" is a finding, not an excuse.
4. Did the data leave your perimeter, and under what agreement?
If your clinical AI uses an external model — Claude, GPT, Gemini, anything hosted outside your environment — the audit shifts to the egress boundary. The auditor will ask which model host received PHI, what business associate agreement governs the relationship, what data residency commitments exist, and whether any prompts crossed a region or jurisdiction boundary. Multi-region deployments under HIPAA and GDPR add layers of complexity here, especially when the model host's infrastructure is itself multi-region.
If you use an on-premises or in-boundary model, the questions are different but no less rigorous. The auditor will ask about the network boundaries, the model's training data lineage, and whether the model's outputs can be traced back to specific inputs in a way that distinguishes hallucination from disclosure.
What the architecture must support: every model call is routed through a gateway that records the model host, the BAA in force, the region of execution, the de-identification applied, and the user and patient context. "We send prompts directly from the application to OpenAI" is a sentence that ends an audit before it begins.
5. Can you reconstruct any single decision the system made?
This question is the audit equivalent of source-code traceability. The auditor picks a single response the agent produced — three months ago, six months ago, a year ago — and asks you to reconstruct it. What was the user's exact question? What retrieval context was assembled? What tools were called and what did they return? What prompt was sent to the model? What response came back? Which parts of the response were surfaced to the user? Were there any human-in-the-loop edits, and what were they?
If you can answer this in an afternoon with a query against your audit store, you are ready. If you need to engage your AI vendor to extract logs, your engineering team to dig through three different stores, and your security team to correlate timestamps, you are not. The retention period is also part of the question — most HIPAA programs require six years, longer in some states for some categories of records.
What the architecture must support: lineage as a first-class data citizen. Every AI output should carry a trace ID that resolves to the full reconstruction of how it was produced. This is not a feature you add in year three. It is the feature you build first.
6. What happens when something goes wrong?
Breach notification is the question that focuses minds. The auditor will ask how you would detect that PHI was disclosed inappropriately by your AI system, how quickly you could identify the affected patients, and how you would notify them under HIPAA's 60-day rule. "Our model hallucinated and we are not sure who saw what" is a breach response that becomes a breach itself.
The harder version of this question concerns inference. If your AI system produced an answer that revealed PHI the user was not authorized to see — not because of a retrieval failure but because the model inferred it from non-PHI context — is that a disclosure? Under most reasonable readings of the Privacy Rule, yes. Designing for that case requires the same logging discipline as the others, plus an evaluation framework that can detect inference leaks before they ship.
What the architecture must support: incident response that begins with a query. Given a suspected disclosure, you must be able to identify the specific records exposed, the users who received the output, the timeframe, and the patients affected, in hours not weeks. This is a function of how cleanly your audit data is structured, not of how skilled your incident responders are.
The Architecture That Answers These Questions
None of these six questions are surprising. The HIPAA rules are public and have not changed materially in years. What is new is that AI systems make answering them harder — because they fragment the access pattern, mediate retrieval through models, generate prompts dynamically, and call out to external services that are not in your direct control.
The architecture that makes these questions answerable rests on four design choices. None are exotic. All are non-negotiable if you intend to operate in regulated clinical environments.
Authorization in the data layer. Every read passes through a single policy engine that knows about user, role, relationship, consent, purpose of use, and minimum necessary. Structured queries, vector retrieval, and agent tool calls are all subject to the same rules and produce the same audit records.
Typed tool interfaces between agents and data. Agents do not write SQL or FHIR search queries. They invoke narrow, audited tools — search_patients, get_observations, semantic_search_notes — each of which inherits the user's permissions, grounds clinical concepts through a terminology service, and writes a record an auditor can read. Letting a model write queries directly is a compliance incident waiting to happen.
Vector storage inside the compliance boundary. Embeddings of clinical notes are PHI, even when the original text is chunked and transformed. They live in storage that meets the same standards as the relational source of truth, with metadata that supports ACL-aware filtering at query time. A third-party vector SaaS is rarely the right answer.
An egress gateway for every model call. All prompts to all model hosts, internal or external, route through a single gateway that handles de-identification, BAA-aware routing, region selection, and token-level logging. The gateway is the only path out of your perimeter, and its logs are the spine of your audit posture.
What an Audit Event Actually Looks Like
The four design choices above land on a simple deliverable: every read, write, agent action, and model call produces one structured audit event. The event records a decision, not a payload. It tells an auditor who did what, to whose data, in what context, and whether it was permitted. Anything beyond that lives in a separate store linked by ID.
A clinician opening a patient's chart and listing the patient's recent observations produces an event that looks like this:
Successful clinical read
event_key = OBSERVATION_LIST
application = EHR
action_type = READ
user_type = CLINICIAN
outcome = SUCCESS
user = jchen.md
tenant = 7b2e9f04-5a31-4d8c-9e72-1c4f8a6d5b29
attributes = [
patient_id = [PT-9182734],
purpose_of_use = [TREATMENT],
resource_count = [47]
]
That is enough to satisfy the first audit question. The auditor knows who the user was, what they did, to which patient, in what application, under what declared purpose, and that the access succeeded. The timestamp is in the event envelope. Forty-seven observations were returned. The minimum-necessary standard is defensible because the purpose of use is recorded; if the purpose were ever set to a non-treatment value, the policy engine would have decided differently.
The event that matters even more is the denial. Most teams forget to emit one. Auditors do not.
Authorization denial — the event most teams forget to emit
event_key = NOTE_READ
application = EHR
action_type = READ
user_type = CLINICIAN
outcome = DENIED
user = jchen.md
tenant = 7b2e9f04-5a31-4d8c-9e72-1c4f8a6d5b29
attributes = [
patient_id = [PT-9182734],
purpose_of_use = [TREATMENT],
deny_reason = [BEHAVIORAL_HEALTH_SEGMENTATION]
]
This event is the proof that your data layer enforced consent. The denial reason names the specific rule that fired. When an auditor asks whether your system protects behavioral health records correctly, you do not show them documentation; you show them the denial events.
When the AI copilot reaches data on the user's behalf, the same event shape covers the access. Two fields establish the chain of accountability: the agent identifier, and the user the agent is acting on behalf of.
Agent acting on behalf of a user
event_key = SEMANTIC_SEARCH_NOTES
application = AI_COPILOT
action_type = READ
user_type = CLINICIAN
outcome = SUCCESS
user = jchen.md
tenant = 7b2e9f04-5a31-4d8c-9e72-1c4f8a6d5b29
attributes = [
patient_id = [PT-9182734],
purpose_of_use = [TREATMENT],
on_behalf_of = [jchen.md],
agent_id = [chart_copilot],
resource_count = [6]
]
The agent retrieved six notes. The user is jchen.md. The on_behalf_of field is also jchen.md. That equality is the proof, recorded in the audit event itself, that the agent did not exceed the user's permissions. If on_behalf_of ever differed from user, or were absent, that is the finding. No prose needed.
Notice what is not in any of these events. There is no prompt content, no token count, no model output, no embedding vector, no hashes. Those belong in a prompt store and an observability store, linked by ID. The audit event records the decision. Mixing the two is the failure mode that produces 50-field audit records that are expensive to store and useless to read.
The Reframe
Most clinical AI teams approach compliance as a layer they add late, often under pressure from a security review or a customer's procurement process. This works in non-regulated AI domains because the cost of getting compliance wrong is reputational. In healthcare, the cost is patient harm, regulatory action, and existential risk to the organization.
The systems that ship and survive are the ones designed, from day one, to answer an auditor's questions in minutes. Authorization, audit, and lineage are not features bolted onto a working system; they are the load-bearing structure of the system itself. Build them first, and the AI fits. Build them last, and the AI does not ship.
If you are responsible for a clinical AI system, the most useful exercise you can do this week is to walk through these six questions for a single response your agent produced last week. If you cannot answer them in an afternoon, you know where the work is.
Top comments (0)