DEV Community

Cover image for Building HealthcareAI with Safe MCP Tooling
Surender Gupta
Surender Gupta

Posted on

Building HealthcareAI with Safe MCP Tooling

#ai #healthcare #mcp #security

AI agents in healthcare should not have unrestricted access to internal systems.

That means no direct access to:

Qdrant
Neo4j
Postgres
Redis
HMS APIs
shell
filesystem
raw logs
secrets
unrestricted HTTP
Enter fullscreen mode Exit fullscreen mode

For HealthcareAI, every capability should go through a narrow, permissioned MCP tool.

Core idea

The architecture should look like this:

AI / CrewAI / Chat UI
        ↓
Permissioned MCP Tools
        ↓
HealthcareAI Systems
Enter fullscreen mode Exit fullscreen mode

The AI layer should never directly touch databases, logs, infrastructure, or hospital management APIs.

The MCP layer becomes the control boundary.

It validates:

schema
role
session authorization
patient scope
risk level
confirmation requirement
redaction
audit logging
Enter fullscreen mode Exit fullscreen mode

Example safe tools

Good MCP tools are domain-specific.

rag.retrieve_context
guidelines.search
guidelines.get_citation
kg.find_condition_candidates
kg.get_condition_neighbors
kg.explain_symptom_links
patient_context.get_deidentified_summary
patient_context.get_encounter_snapshot
hms.appointment.create_draft
hms.appointment.confirm_and_book
clinical.prescription.create_draft
clinical.prescription.safety_check
clinical.prescription.submit_after_doctor_confirmation
observability.get_trace_summary
devops.get_service_health
Enter fullscreen mode Exit fullscreen mode

These tools expose controlled capabilities instead of raw system access.

Bad tools to avoid

Avoid tools like:

run_sql
run_cypher
execute_shell
fetch_any_url
read_any_file
query_qdrant_raw
access_logs_raw
Enter fullscreen mode Exit fullscreen mode

These are too broad.

They are difficult to permission, difficult to audit, and dangerous in a healthcare environment.

Risk levels matter

Different HealthcareAI tools should have different risk levels.

Low       → observability summaries
Medium    → RAG retrieval, guideline lookup, KG search
High      → appointment booking
Critical  → prescription-like workflows, infra actions
Enter fullscreen mode Exit fullscreen mode

The tool design should reflect the risk.

Read-only tools may only need authorization.

Write tools should require confirmation.

Critical clinical actions should require doctor confirmation.

Draft-confirm-execute pattern

For write workflows, HealthcareAI should use:

draft → confirm → execute
Enter fullscreen mode Exit fullscreen mode

Example appointment flow:

hms.appointment.create_draft
        ↓
user confirms doctor, facility, slot, mode
        ↓
hms.appointment.confirm_and_book
Enter fullscreen mode Exit fullscreen mode

Example prescription flow:

clinical.prescription.create_draft
        ↓
clinical.prescription.safety_check
        ↓
doctor confirmation
        ↓
clinical.prescription.submit_after_doctor_confirmation
Enter fullscreen mode Exit fullscreen mode

AI can draft.
AI can assist.
AI should not silently execute critical healthcare actions.

Example audit event

Every MCP call should produce an audit log.

{
  "event": "rag.retrieve_context",
  "query_hash": "...",
  "user_role": "doctor",
  "chunks_returned": 8,
  "sources": ["ICMR guideline"],
  "trace_id": "..."
}
Enter fullscreen mode Exit fullscreen mode

For patient context, sensitive fields should be redacted.

{
  "event": "patient_context.get_deidentified_summary",
  "patient_ref_hash": "...",
  "encounter_id": "enc_123",
  "fields_returned": ["age", "gender", "allergies", "vitals"],
  "fields_redacted": ["phone", "email", "address"],
  "trace_id": "..."
}
Enter fullscreen mode Exit fullscreen mode

Audit logs make the system traceable and reviewable.

MCP tool checklist

Before adding a HealthcareAI capability, check:

✅ Server owner is clear
✅ Input/output schema is strict
✅ No raw SQL/Cypher/shell/filesystem/unrestricted HTTP
✅ Permission rule exists
✅ Patient scope is enforced
✅ Risk level is assigned
✅ Confirmation rule is explicit
✅ Audit log is mandatory
✅ Data is redacted or minimized
Enter fullscreen mode Exit fullscreen mode

Final rule

HealthcareAI should not be “AI with system access.”

It should be:

AI with controlled responsibility
Enter fullscreen mode Exit fullscreen mode

If a capability cannot be safely expressed as:

MCP server
+ tool/resource/prompt
+ schema
+ permission
+ confirmation
+ audit
Enter fullscreen mode Exit fullscreen mode

then it should not be implemented yet.

That is the foundation for safer healthcare AI architecture.

Top comments (0)