Originally published at krasyn.com/blog/ai-primary-care-hype-vs-reality
The AI in healthcare space is crowded with vendor claims and real clinical research in roughly equal measure. This evidence-based guide separates the AI applications that have demonstrated real-world impact from those that remain aspirational in primary care.
How to Think About AI Claims in Healthcare
Every EMR vendor, health system, and technology startup has an AI story in 2026. The claims range from "faster note generation" (measurable, testable) to "improves clinical decision quality" (complex, context-dependent) to "transforms patient outcomes" (unmeasured, aspirational).
Three questions to ask about any AI claim:
- Is this published in a peer-reviewed journal, or is it from a vendor white paper?
- What is the control condition? (AI vs. no AI, or AI vs. current best practice?)
- Was this studied in a setting similar to mine (primary care, outpatient, similar volume)?
What Works: AI Applications with Published Evidence
1. Ambient AI Documentation (Strong Evidence)
Multiple peer-reviewed studies (JAMA Network Open 2023, NEJM Catalyst 2024, Health Affairs 2024) demonstrate:
- Documentation time reduction: 25-50% per encounter
- After-hours documentation decrease: 30-40%
- Physician-reported note quality: equivalent or better than self-authored notes in 80-90% of encounters
- Burnout score improvement: 15-25% on validated instruments in 6-month studies
What to look for: The AI listens passively during the encounter and produces a draft note that the physician reviews and signs. No dictation script, no command words, no changing how you speak to patients.
2. AI-Assisted Coding and Billing Review (Moderate-Strong Evidence)
- A 2024 University of Michigan study found 8-12% additional annual revenue per physician from AI coding review
- Claim denial rate reductions of 30-50% when AI flags documentation gaps before submission
- HCC capture improvements of 15-25% for Medicare Advantage panels
3. Medication Interaction and Dosing Alerts (Well-Established)
Rule-based drug interaction checking has been in EMRs for 20+ years. AI-enhanced alerting that learns which alerts a given physician acts on and suppresses those routinely overridden is associated with substantially lower override rates.
4. Sepsis and Deterioration Early Warning (Hospital Setting; Limited Outpatient Evidence)
AI-based early warning systems have demonstrated reduced sepsis mortality in inpatient settings. If an outpatient AI vendor claims their system prevents hospitalizations, ask for the study design and control condition before accepting the claim.
What Remains Aspirational: Where Evidence Is Weak or Absent
AI-Assisted Differential Diagnosis
In undifferentiated primary care presentations -- the actual environment -- the published performance is modest and often not better than a well-structured clinical reasoning process.
AI-Generated Patient Communication at Scale
A 2023 JAMA study found that AI-generated patient instructions were accurate in 78% of cases -- meaning 22% contained clinically significant errors requiring correction. AI-generated patient communication without mandatory physician review is a patient safety risk in its current state.
Predictive Analytics for Population Health
Models work at population level (they correctly identify higher-risk populations) but individual-level predictions remain imprecise. These should inform panel-level intervention prioritization, not individual care decisions without clinical assessment.
Evaluating AI Claims from EMR Vendors
| Claim Type | What to Ask | Red Flags |
|---|---|---|
| "Reduces documentation time by X%" | What was the baseline? Measured how? In what setting? | Only white paper evidence; no peer-reviewed study |
| "Improves clinical decision quality" | Compared to what? In what population? | Claims based on narrow disease populations |
| "AI meets HIPAA requirements" | Where is audio/data processed? Who has access? | Vague answers about data residency; no BAA offered |
| "Reduces burnout" | What validated instrument? Over what time period? | Testimonials instead of validated measures |
Why Transparency Matters
The AI tools worth trusting are the ones that show their work. That means showing the physician which note text supported a billing code suggestion, flagging when the AI is uncertain, and maintaining a documented list of what the AI cannot do reliably.
Originally published at krasyn.com/blog/ai-primary-care-hype-vs-reality
Top comments (0)