I sat through nine AI-QMS demos. Only one made the tradeoffs visible.

We are mid-evaluation on AI tools for our QMS, and I have sat through more vendor demos in the last six weeks than I want to admit. Most of them were glossy. Most of them were also useless, in the specific sense that they did not help me decide whether to buy.

Here is what the average demo does. It shows a chatbot answering a generic regulatory question. It shows a "draft a CAPA" button that produces a confident-sounding paragraph. It shows a dashboard with red, amber, and green tiles. None of that tells me whether the tool will survive a notified body audit, integrate with our existing change control, or stop a junior engineer from publishing an unverified root cause analysis into a Technical File.

To be fair, vendors are responding to a market signal. Everyone is being told they need AI, and AI is being equated with chat interfaces. The harder question — does the tool actually link to my controlled documents, my risk files, my post-market data — does not make for a good slide.

The first piece that made me stop and reread

Then I came across a vendor-adjacent article on AI for QMS in medical devices (qmswrapper.com/ai-qms-for-medical-devices, if the link holds). It was the first thing I had read from a vendor-adjacent source that named the tradeoffs instead of dodging them.

The framing that landed for me was around reviewability and traceability — the idea that AI assistance in a QMS is only useful if every output is reviewable, every source is traceable, and the workflow connects back to your existing controlled processes. Not magic. Native workflow integration.

Granted, that is the pitch any sensible vendor should be making. But what made this one readable was that it conceded the hard parts: what the AI should not do, where human sign-off is non-negotiable, and why "automated CAPAs" is a phrase that should make any RA person nervous unless the automation is narrow, scoped, and auditable.

In practice this means: if a tool promises to "auto-close CAPAs" or "draft your PSUR", the next question is not how clever it is. The next question is what stays under human control, what is logged, and whether the system can show a notified body the entire reasoning chain six months later.

What the useful demos actually show

After nine demos, the ones that earned a second meeting had three things in common.

• They showed the failure mode. A good demo includes an example where the AI gets it wrong, and shows you how the system handles the correction. If a vendor cannot show me a wrong answer, I do not trust their right ones.
• They named the regulatory anchor. Per Article 10(9) of MDR, the manufacturer is responsible for the QMS output regardless of who — or what — produced it. Tools that acknowledge this and design around it — controlled assistance, safe assistance for CAPAs, explicit human-in-the-loop gates — are different from tools that treat regulatory language as decoration.
• They talked about CAPA-driven risk assessment. Not as a feature, but as a workflow: when a CAPA opens, the risk file updates; when a risk changes, the change control evaluates downstream impact; when a change is approved, training and document control follow automatically. That is a connected workflow. Most demos showed me isolated screens.

The five questions I am now asking every vendor

I am not going to publish the full scoring rubric, but here are the questions that consistently separate the genuine tools from the slideware.

1. Where does the AI output live before a human signs off — and can an auditor see its full history?
2. Which decisions are blocked until a qualified person approves them, by configuration rather than convention?
3. How does the tool link a CAPA to the affected risk file, change record, document, and supplier record — natively, not by export-and-import?
4. What happens when the underlying regulation changes? Does the AI update its references, or do I?
5. Show me a CAPA you helped close last quarter. Show me the audit trail. Show me who signed.

If a vendor can answer all five with screenshots rather than promises, I will sit down again. If they deflect, I will politely move on.

What this means for our evaluation

We are not buying an AI. We are buying a controlled, reviewable, traceable layer on top of the QMS we already have. The framing that helped me most was the one that treated AI as assistance — scoped to specific tasks, with the human in the loop and the auditor's question in mind.

The tradeoffs are not new. They are the same tradeoffs we have always made about process automation in regulated environments. The new part is that vendors are finally being asked to be specific about them. That is a useful development, regardless of which tool wins our evaluation.

For what it is worth, the qmsWrapper piece also pointed out that their QMS Manual is integrated throughout the software — meaning the executed workflows map back to documented procedures rather than living in a separate system. That detail mattered to me more than the AI pitch itself, because it is the bit that breaks under audit pressure when the two systems drift apart.

Question for readers: when you evaluated AI tools for your QMS, which single question cut through the marketing fastest? I am still collecting, and the five above are not the last word.