If you're building an AI agent for a regulated domain, the hard part isn't the model — it's everything around it. Here are five things I believed (or heard) that turned out to be wrong once real PHI and a real audit were involved.
1. "RAG is enough to make it safe." Retrieval grounds the answer, but safety comes from the gate: a below-confidence or high-stakes result must halt and route to a human, not auto-resolve. Retrieval + a real human-in-the-loop interrupt, not retrieval alone.
2. "PHI handling is a compliance checkbox at the end." It's an input-path concern. Detect and redact PHI before the prompt is built, so the model reasons over de-identified facts. Bolting it on afterward means it already leaked.
3. "Autonomy is the goal." In regulated work, full autonomy is a liability. Model the human approval step as a first-class state in your workflow (a durable interrupt you can resume), not an afterthought.
4. "Logging is just for debugging." Your audit trail is a product feature. Append-only, capturing request → retrieval → decision → citation → sign-off. It's the difference between "trust us" and an exportable record when compliance asks.
5. "An agent is a prompt." It's a workflow: retrieval, tool calls across systems, a control plane (we use an MCP-style layer), confidence thresholds, and escalation. The prompt is maybe 5% of it.
The theme: governance is the architecture, not a wrapper. We ship this as governed healthcare AI agents at IntelliBooks Studio — more at intellibooks.ai/overview.
Top comments (0)