DEV Community: wang zhengpeng jay

How I Built a Self-Hosted Family AI Health Steward (Your Health Data, on Your Shelf)

wang zhengpeng jay — Tue, 11 Aug 2026 03:14:41 +0000

TL;DR — I built and open-sourced AI Health Steward, a self-hosted, private AI health manager for families. It reads photos of lab reports with multimodal LLMs, builds a structured per-person health profile, shows trends on a dashboard, and answers health questions grounded in your actual data — all running on your own server. Privacy isn't a feature; it's the whole point. Star it on GitHub.

The problem: your health data is a product

Every family has a shoebox — or a folder — of medical reports: blood tests, blood-pressure logs, prescriptions, scan findings. And every "convenient" health app wants to hold those records for you. But hold them where? On someone else's cloud, to be monetized, analyzed, or lost when the startup pivots.

Health records are the most sensitive data you own. They shouldn't be a product. They should live on your shelf.

So I built the opposite: a self-hosted AI health steward where the data never leaves your server.

What it does

📄 Take a photo of a lab report → structured data. A multimodal LLM extracts key metrics (BP, glucose, lipids, CBC…) with your confirmation before anything is filed.
🧬 A person-level health profile as the single source of truth — basics, metrics, diagnoses, medications, allergies, lifestyle, family history, and data provenance (where each value came from).
📈 Trend visualization with anomaly markers and clinical critical-value alerts (e.g. BP ≥ 180/110 triggers a "see a doctor" banner).
💬 AI consultation grounded in real data — not a generic chatbot. Intent routing + function calling means answers reflect your profile, not Wikipedia.
🗓️ Personalized checkup plans via a 1+X+Y framework, with budget tiers and safety/contraindication screening.
📋 Periodic health summaries (weekly/monthly/yearly), risk scales (PHQ-9, GAD-7, diabetes, ASCVD), and follow-up/medication reminders.
🧠 RAG over your own history — archived reports are vectorized so you can ask "what did my A1C trend look like over 3 years?"

The architecture

┌──────────────┐     ┌──────────────────────────────────────┐
│   WebUI      │────▶│            FastAPI backend           │
│ React + Vite │     │  API / services / providers / prompts │
└──────────────┘     └───────────────┬──────────────────────┘
┌──────────────┐                     │
│ Feishu bot   │────▶  WebSocket ◀───┤
└──────────────┘                     ▼
                     ┌──────────────────────────────────────┐
                     │  PostgreSQL 16 + pgvector (RAG)      │
                     └──────────────────────────────────────┘
                     ┌──────────────────────────────────────┐
                     │  Model providers (pluggable)         │
                     │  OpenAI-compatible API / Ollama      │
                     └──────────────────────────────────────┘

Backend: Python 3.12 + FastAPI, SQLAlchemy 2.0 + Alembic
Frontend: React 18 + Vite + TypeScript + TailwindCSS
Data: PostgreSQL 16 + pgvector (for report semantic search)
AI: pluggable OpenAI-compatible models (multimodal/text) + optional local Ollama for fully-offline operation
Deploy: single docker compose up -d

Why FastAPI + React?

FastAPI gives clean async handlers and Pydantic-validated schemas — perfect for the AI-agent-style tool-calling layer. React + Vite keeps the dashboard snappy. PostgreSQL + pgvector avoids a second vector database — one storage engine for structured data and embeddings keeps the deploy story simple for a home server.

Key engineering decisions

1. The "person-level profile" as a single source of truth.
Every extracted value carries provenance — which report, which date, confirmed or not. The AI consultation layer reads from this profile rather than re-interpreting raw uploads every time. This is what makes answers grounded instead of hallucinated.

2. Cost is a real concern for a home app.
I aggressively cut LLM calls: no duplicate metric-extraction calls per conversation, and periodic summaries skip the LLM entirely when a period had no new data. A self-hosted app that costs pennies to run actually gets used.

3. Structured output + human confirmation.
AI extraction is powerful but not infallible. Every report goes through extract → confirm → archive, so garbage never silently enters the health record.

4. Privacy as the default posture.
Data stays local. Model calls go only to the provider you configure, and you can go fully offline with Ollama. Optional Bearer-token auth + per-member rate limiting protects the instance.

The dual-entry UX (Web + chat)

A full dashboard is powerful but heavy for "hey, is this blood pressure okay?" So there are two entrances:

WebUI — the complete management backend: profile dashboard, report management, trends, checkup recommendations, summaries, settings.
Feishu/Lark bot — the lightweight daily entry: snap a report photo, ask a quick question, log a metric — all in a chat you already have on your phone.

Data flows between both automatically. Same profile, same single source of truth.

What I'd tell my past self

Build the privacy story first, not last. Self-hosted users care about why — lead with it.
A photo of a report is the killer feature. Multimodal extraction is what separates this from a spreadsheet.
Keep it runnable on 2 GB RAM. The easiest project to promote is the one people can actually deploy on a spare mini-PC.
Ship a bilingual README (English + Chinese) — the self-hosting community is global.

Try it

git clone https://github.com/wangzhengpengjay/AI-Health-Steward.git
cd AI-Health-Steward
cp .env.example .env        # set MULTIMODAL_API_KEY and TEXT_API_KEY
cp .env backend/.env
docker compose up -d
docker exec health-steward-backend alembic upgrade head
# WebUI: http://localhost:5173   |  API docs: http://localhost:8000/docs

A one-command demo-data seed makes it easy to explore before wiring up real accounts.

If this resonates, give it a ⭐ — it helps other people who want their health data on their own shelf find it. And PRs and feature suggestions are genuinely welcome.

AI Health Steward is an open-source project (MIT). It is not a medical device, does not provide diagnoses, and is not a substitute for professional medical care.