DEV Community

Cover image for I Built a Privacy-First Health Record MCP Server That Runs Entirely on Your Machine
Priyanshu Goel
Priyanshu Goel

Posted on

I Built a Privacy-First Health Record MCP Server That Runs Entirely on Your Machine

#mcp #ai #python #healthtech

How I used the Model Context Protocol, Gemini Vision, and AES-256 encryption to let Claude answer questions about your personal health data — without sending that data anywhere.

The Problem With Health AI

Every major health AI product has the same architecture: you upload your documents to their cloud, they process and store them, and you hope their privacy policy holds up. For general wellness tips, that's probably fine. For your actual medical records — prescriptions, lab results, discharge summaries — it's a meaningful trust ask.

I wanted something different. I wanted Claude to be able to answer questions like "what medications am I on?" or "is it safe to add Ibuprofen to my current medications?" using my actual health records, without those records ever leaving my machine.

The result is MedMemory — an MCP server that stores your health data in an AES-256 encrypted SQLite database locally, extracts structured information from medical documents using Gemini Vision, and exposes 8 tools to Claude Desktop so you can query your health history in natural language.

What is MCP?

The Model Context Protocol (MCP) is Anthropic's open standard for giving LLMs access to external tools and data sources. Instead of fine-tuning a model or building a RAG pipeline, you write a server that exposes typed tools — functions with names, descriptions, and input schemas — and the LLM decides when to call them based on the conversation.

The key insight is that the tool docstrings are the intelligence layer. When you ask Claude "am I up to date on my vaccines?", Claude doesn't know it has a vaccination tool until it reads the docstring:

@mcp.tool()
async def get_vaccination_status(ctx: Context) -> dict:
    """Returns all recorded vaccinations and flags overdue or missing ones.

    Use this when the user asks about:
    - their vaccination history or immunisation records
    - whether they are up to date on vaccines
    - what vaccines they are missing or overdue for
    - travel vaccine recommendations
    """
Enter fullscreen mode Exit fullscreen mode

That docstring is what makes Claude call the right tool at the right time. Writing good docstrings is the most important skill in MCP development.

The Architecture

MedMemory has three layers:

1. Storage — SQLCipher encrypted SQLite

All health data lives in a local SQLite database encrypted with SQLCipher (AES-256). The encryption key is set by the user on first run and never leaves the machine. If you open the database file in any text editor, you see random bytes — not readable SQL.

$ xxd medmemory.db | head -3
00000000: e7ea 869f 47b1 bec2 8eb8 78f3 8fe9 57e7  ....G.....x...W.
00000010: 44c2 3e72 3056 bb14 ae19 1b53 5530 54e9  D.>r0V.....SU0T.
Enter fullscreen mode Exit fullscreen mode

The migration from plain SQLite to SQLCipher was exactly 3 lines of code — import sqlcipher3 as sqlite3, plus two PRAGMA lines in get_connection(). Everything else stayed identical. SQLCipher is a true drop-in replacement.

2. Ingestion — Gemini Vision for real-world documents

Medical documents in the real world are messy — handwritten prescriptions, scanned PDFs, photos taken at angles. I started with Tesseract OCR but it failed completely on handwritten Indian prescriptions. The fix was bypassing OCR entirely for image-based documents and sending the image directly to Gemini Vision, which reads handwriting natively.

The extraction pipeline now works like this:

  • Native text PDF (e.g. digital lab report): PyMuPDF extracts markdown → Gemini structures the entities
  • Scanned/handwritten PDF or image: render to PNG → send directly to Gemini Vision

The extraction prompt includes explicit brand-to-generic drug name mappings and lab marker normalisation rules, plus a few-shot example showing exactly what output format is expected. This dramatically improved accuracy on real documents:

"Glycomet" → "Metformin"
"Storvas" → "Atorvastatin"  
"Gabantin-GRS" → "Gabapentin"
"Zerodol-P" → split into "Aceclofenac" and "Paracetamol"
"LDL Cholesterol" → "LDL"
"S. Creatinine" → "creatinine"
Enter fullscreen mode Exit fullscreen mode

3. MCP Tools — 8 tools covering the full health record

Tool What it does
ingest_health_document Parse any medical PDF or image into structured DB records
get_current_medications Return active medication list with doses and frequencies
get_lab_trend Show historical readings for any lab marker
get_visit_history Doctor visits with optional specialty filter
get_vaccination_status Cross-reference DB against WHO adult schedule, flag gaps
check_drug_interaction Query OpenFDA API against your current medication list
get_allergies Return recorded allergies
generate_health_summary Synthesise all tables into a printable one-pager

The Drug Interaction Tool — The Most Clinically Important Feature

check_drug_interaction is the tool I'm most proud of. It takes a newly prescribed drug name, pulls your current medications from the encrypted DB, queries the OpenFDA drug label API for the new drug's interaction data, and returns a structured result combining both.

# Claude Desktop conversation
User: My doctor just prescribed Ibuprofen for back pain. 
      Is it safe with my current medications?

Claude: [calls get_current_medications → Metformin, Atorvastatin, Aspirin]
        [calls check_drug_interaction("Ibuprofen") → FDA label data]

        ⚠️ Important concern — Ibuprofen + Aspirin:
        Both are NSAIDs. Taking them together significantly increases 
        the risk of stomach bleeding. The FDA specifically warns that 
        taking multiple NSAIDs at the same time raises this risk.

        Metformin + Ibuprofen: Both can affect kidney function. 
        Monitor kidney health if you proceed.

        Consider Acetaminophen as a safer alternative for back pain.
Enter fullscreen mode Exit fullscreen mode

A generic AI without your records would give the same NSAID warning to everyone. This one knows you're already on Aspirin and gives you a personalised, clinically relevant answer. That's the difference between a generic chatbot and a tool that actually knows your situation.

The Privacy Architecture

Here's exactly what goes where:

Stays on your machine:

  • The SQLite database (encrypted with your passphrase)
  • Your encryption key (stored in local .env, never transmitted)
  • All medications, lab results, visits, vaccinations, allergies

Goes to Gemini API (during ingestion only):

  • The extracted text or image from the document you're ingesting
  • This is a transient API call — Gemini doesn't store it in your health record

Goes to OpenFDA API:

  • Only the drug name you're checking — no personal data

Goes to Anthropic (same as any Claude conversation):

  • The tool call results that Claude uses to answer your question
  • This is the same data that flows in any Claude Desktop conversation

You can verify the encryption yourself:

xxd ~/medmemory.db | head -3
# Should show random bytes, not "SQLite format 3"
Enter fullscreen mode Exit fullscreen mode

What I Learned Building This

1. Docstrings are the product. The quality of your tool docstrings determines whether Claude calls the right tool at the right time. I spent as much time on docstrings as on the actual tool logic.

2. Test on real documents early. I built the ingestion pipeline on clean synthetic PDFs and it looked great. When I tested on a real handwritten prescription from a neurologist in Ghaziabad, it extracted nothing. Real-world documents are fundamentally different from test data.

3. SQLCipher is genuinely a 3-line migration. I expected encrypting a SQLite database to be a multi-day project. It was import sqlcipher3 as sqlite3 plus two PRAGMA lines. The API is identical. The hardest part was getting the Python package to compile on Apple Silicon.

4. MCP is architecture, not a feature. The same DB helpers that power the MCP tools also power a FastAPI bridge that serves a Next.js companion UI. The DB layer is the business logic. MCP and FastAPI are just two different ways to expose it.

5. OpenFDA is free and powerful. The drug interaction data comes from the same FDA drug label database that pharmacists use. No API key required for low volume, and the data is authoritative.

Try It 

pip install medmemory-mcp
medmemory-setup
Enter fullscreen mode Exit fullscreen mode

Then add to your Claude Desktop config:

{
  "mcpServers": {
    "medmemory": {
      "command": "medmemory-server"
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

Or try the hosted demo (synthetic patient data, no real health records):

{
  "mcpServers": {
    "medmemory-demo": {
      "command": "npx",
      "args": ["-y", "mcp-remote", 
               "https://web-production-ba446.up.railway.app/sse"]
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

Links:

Built with Python, FastMCP, SQLCipher, Gemini Vision, OpenFDA, FastAPI, and Next.js. Questions or feedback welcome in the comments.

Top comments (0)