How Hindsight Turned Fragmented Visit Data Into a Usable Timeline
Seven visits. Seven Firestore documents. Seven correct AI summaries. Zero sense of what had changed across them — and no way to tell without reading all seven in order.
What MediPass Does and How It Fits Together
MediPass is a digital health passport built around the physical workflow of a hospital outpatient department. It doesn't try to replace the paper prescription or the face-to-face consultation. It wraps around them.
The flow is concrete: a patient arrives, shows their phone number at the reception desk, a receptionist verifies identity via OTP and generates a transactional visit number, the patient enters the doctor's queue, the doctor runs a read-only consultation, the receptionist uploads the paper prescription and any new lab reports at checkout, and the patient's phone gets a plain-English AI explanation of every document the moment it lands. That's the loop. It closes at every visit without requiring anyone to change how they already work.
Four roles, four portals, one Firestore database. The stack: React 18 with TypeScript, Vite, Firebase Firestore with real-time onSnapshot listeners, Firebase Auth phone OTP, Cloudinary for document CDN, and Groq's Qwen 27B vision model for multimodal document parsing. No backend server — Firestore security rules enforce role isolation at the database layer. The doctor portal has no search bar, no upload button, no edit access. Those aren't hidden UI elements; they're absent because the underlying Firestore rules reject any write from a doctor role except the single narrow status transition that marks a session as consulted.
The Firestore schema has one document per uploaded file in the records collection:
// lib/firestore.ts — records collection interface
interface MedicalRecord {
patientId: string;
sessionId: string;
hospitalId: string;
uploadedByStaffId: string;
type: "prescription" | "lab_report" | "imaging_report" | "discharge_summary";
sourceFileUrl: string; // Cloudinary CDN URL
extractedText: string; // Tesseract.js OCR output
aiExplanation: string; // patient-facing plain-English summary
vitals: {
bloodPressure: string | null;
heartRate: number | null;
temperature: number | null;
spo2: number | null;
weight: number | null;
};
createdAt: Timestamp;
}
The aiExplanation field is the output of the Groq vision pipeline — an accurate, plain-English description of whatever document was uploaded during that visit. The problem I'm about to describe is not about that field being wrong. It's about what that field can't tell you on its own.
The Fragmentation Problem
Every records document in Firestore is a correct, self-contained description of one document from one visit. That's exactly what you want from a per-document store. It's also exactly the wrong unit of information for answering clinical questions about a patient's trajectory.
A patient with seven OPD visits for a chronic condition — hypertension managed with a beta-blocker, quarterly BP readings, a medication adjustment at visit four — has seven records documents. Each one has an accurate aiExplanation field. What those seven fields don't contain, in aggregate, is the answer to the question a patient actually wants answered when they open their dashboard: What's changed? Is my condition improving? Am I still on the same medication? When did that change?
My first attempt to solve this was naive: load the last N records, concatenate their aiExplanation fields into a single string, pass it to the model, ask it to synthesize a longitudinal summary. It worked for N=2. At N=5 it started producing vague output. At N=7 across a patient with multi-page lab reports and prescription uploads mixed together, it hit token limits and the model spent its capacity organizing the data dump rather than reasoning over it. The output was technically correct and clinically useless.
The problem with concatenation-as-context is that you're handing the model raw chronological material and asking it to do the distillation work at read time, for every patient, on every dashboard load. That doesn't scale — not in tokens, not in latency, not in coherence.
The real problem was architectural: I had invested nothing at write time and was paying for it entirely at read time.
How Hindsight Changed the Write-Time Investment
The framing that shifted my thinking came from reading about structured agent memory as a first-class architectural concern rather than a retrieval optimization. The distinction is this: retrieval assumes the source documents are the right unit to query against. Memory assumes you need a separate, maintained representation of what's clinically significant — updated incrementally as new information arrives, structured by category, queryable by recency — so that reads are targeted and fast rather than exhaustive and slow.
Hindsight implements this as a namespace-per-entity model. Each patient gets a memory namespace. Clinical facts extracted from new records are upserted into that namespace with source record references and timestamps. Prior values for the same fact aren't overwritten — they're preserved with their timestamps, building a time series that the dashboard can query directly rather than reconstructing from raw document history.
The Hindsight documentation describes the upsert semantics clearly: a fact with a matching key gets a new entry appended to its history rather than the old entry replaced. For medication tracking, that means a medication change at visit four doesn't erase the original prescription from visit one — it appends a new entry with the updated dosage and a pointer back to the visit four record. The full change history is in the namespace, queryable as a time series.
I inserted the memory extraction step into the upload pipeline between the AI explanation call and the Firestore write:
// lib/uploadPipeline.ts — the write sequence
async function processUpload(
file: File,
context: UploadContext
): Promise<void> {
// 1. Upload to Cloudinary — real URL before anything proceeds
const sourceFileUrl = await uploadToCloudinary(file);
// 2. OCR (Tesseract.js, runs client-side in a Web Worker)
const extractedText = await runOCR(file);
// 3. AI explanation — Groq Qwen 27B via API
const aiExplanation = await runAIExplainer(extractedText, context.recordType);
// 4. Memory extraction — upsert facts to Hindsight before Firestore write
const recordId = doc(collection(db, "records")).id;
await hindsight.memory.upsert({
namespace: `patient:${context.patientId}`,
facts: extractClinicalFacts(aiExplanation, context.recordType, recordId),
});
// 5. Firestore write — last, after all upstream steps confirmed
await setDoc(doc(db, "records", recordId), {
patientId: context.patientId,
sessionId: context.sessionId,
hospitalId: context.hospitalId,
uploadedByStaffId: context.staffId,
type: context.recordType,
sourceFileUrl,
extractedText,
aiExplanation,
vitals: context.vitals,
createdAt: serverTimestamp(),
});
}
extractClinicalFacts parses the already-generated aiExplanation — not the raw OCR text — and returns a typed array of facts by category: medications, vitals_trend, lab_trend, allergies, diagnoses. Each fact carries the recordId as a source reference so the timeline can link back to the original document.
The Firestore write is last because a record document with a missing sourceFileUrl or empty aiExplanation is worse than no record. The memory upsert happens before the Firestore write so the two are in sync: by the time the record is visible to the patient's dashboard listener, the memory namespace already reflects its contents.
Querying the Timeline
On the patient dashboard, the context for the longitudinal timeline panel comes from two parallel reads:
// lib/patientContext.ts — parallel reads at dashboard load
export async function buildPatientDashboardContext(patientId: string) {
const [recentRecords, patientMemory] = await Promise.all([
// Firestore: last 3 records for the current visit detail cards
getRecentRecords(patientId, { limit: 3 }),
// Hindsight: full structured memory for the longitudinal timeline
hindsight.memory.query({
namespace: `patient:${patientId}`,
categories: ["medications", "vitals_trend", "lab_trend", "allergies", "diagnoses"],
recencyWeight: 0.65,
limit: 40,
}),
]);
return {
recentRecords,
timeline: buildTimeline(patientMemory.facts),
};
}
function buildTimeline(facts: HindsightFact[]): TimelineEntry[] {
// Group by category, then sort each group by recordedAt ascending
// to produce per-category time series suitable for trend display
const grouped = groupBy(facts, (f) => f.category);
return Object.entries(grouped).map(([category, entries]) => ({
category,
entries: entries
.sort((a, b) => new Date(a.recordedAt).getTime() - new Date(b.recordedAt).getTime())
.map((e) => ({ ...e, sourceUrl: resolveRecordUrl(e.sourceRecordId) })),
}));
}
The recentRecords from Firestore feed the current visit panel — the detailed per-document cards at the bottom of the dashboard. The timeline from Hindsight feeds the longitudinal summary panel at the top. They answer different questions and are built from different sources. Neither read blocks the other.
recencyWeight: 0.65 on the Hindsight query means more recent facts rank higher in the results, but older facts — a drug allergy documented two years ago, a chronic diagnosis from the first visit — aren't excluded. A static allergy flag is more important than a recent-but-normal blood pressure reading. The weighting keeps both in context without requiring hand-tuning of which specific facts to always include.
What the Patient Actually Sees
For the hypertension patient with seven visits, the timeline panel renders something like this above the individual visit cards:
📋 Health Timeline
Diagnoses
• Essential hypertension — since Jun 2024 (Visit 1)
Medications
Jun 2024 Amlodipine 5mg — once daily
Sep 2024 Amlodipine 5mg — once daily [no change]
Dec 2024 Amlodipine 10mg — once daily [dose increased ↑]
Mar 2025 Amlodipine 10mg + Telmisartan 40mg [second agent added]
Jun 2025 Amlodipine 10mg + Telmisartan 40mg [no change]
BP Trend (last 7 visits)
Jun → Sep → Dec → Mar → Jun → Sep → Dec
158/98 → 152/94 → 148/90 → 142/88 → 136/84 → 132/82 → 130/80
↓ controlled improvement
⚠️ Flags
• Allergy: ACE inhibitors (documented Sep 2024 — persistent)
• Creatinine borderline elevated at last visit [VERIFY WITH DOCTOR]
That panel is built entirely from the Hindsight memory namespace — no Firestore reads, no re-summarization, no token cost beyond the initial writes at upload time. The medication history is a time series of medication facts, each with a timestamp and a link back to the source record. The BP trend is a time series of vitals_trend facts. The allergy flag is a allergies fact from visit two that has been present in the namespace ever since, surfacing at the top of the context because persistent flags rank high regardless of recency weighting.
Below the timeline panel, the last three visit records render their full per-document AI explanations as detail cards. A patient who wants to read exactly what was in last month's prescription can tap through to the original document — the sourceUrl in each timeline entry links back to the Cloudinary CDN URL stored on the records document.
Lessons Learned
Invest at write time or pay at read time. Concatenating past summaries into a context window at read time is the deferred cost version of the same work. It's expensive in tokens, slow for patients with long histories, and produces unfocused output because the model is organizing data rather than reasoning over it. Write-time extraction from the already-generated AI explanation is cheap; it runs against structured text, not raw documents. The per-patient cost is paid once per upload and amortized across every subsequent dashboard load.
Memory and documents are different data structures for different questions. A Firestore records document answers: what was in this specific file on this specific date? A Hindsight memory namespace answers: what is true about this patient across their clinical history? Those are different questions that need different data structures. Building one to do the work of both produces a system that technically has all the data and practically can't surface the right context.
The Firestore write should always be last. Without a backend transaction that wraps the full pipeline, write order is your consistency boundary. If the OCR step fails after the Cloudinary upload but before the Firestore write, the patient has no record of the upload in their timeline — that's fine, the file exists in Cloudinary and the receptionist can retry. If the Firestore write succeeds before the Hindsight upsert, the record exists in the patient's dashboard but isn't reflected in their longitudinal timeline — that's a silent inconsistency that's much harder to recover from. Memory upsert before Firestore write, always.
Time series needs source references to be navigable. The first version of the Hindsight upsert didn't carry sourceRecordId on each fact. The timeline showed a BP trend but there was no way to navigate from a data point in the trend back to the original document. Adding sourceRecordId to every fact at write time — before the Firestore write, so the ID is available — turned the timeline from a read-only display into a navigable history. A patient who sees an unexpected value in their trend can tap it and read the original document it came from.
Recency weighting is not recency filtering. A drug allergy flag from three years ago is more important than a normal blood pressure reading from last week. Any memory query for clinical context needs to keep both in scope — recent events for current context, persistent flags regardless of age. The recencyWeight parameter on the Hindsight query does this without hand-coding which specific fact categories to always include; it weights recent facts higher without excluding old ones, and persistent flags naturally rank high because they're present in every query result.
The seven visit records were accurate from the start. What was missing wasn't accuracy — it was continuity. Getting continuity required a different write path, a different query, and a different data structure from the one Firestore naturally provides. The Firestore records didn't change; they're still there, still correct, still the source of truth for what was in each document. What changed was the layer that sits between those records and the question a patient actually wants answered when they open their dashboard.
Top comments (0)