The first architecture question we faced was not which model to call. It was what the agent should be allowed to remember, and at what scope.
Two kinds of memory Clinic Desk uses Hindsight as a persistent memory layer, but it does not treat all memories as interchangeable. We designed two scopes: patient memory and clinic memory. That distinction became one of the most useful product decisions because it mirrors how the application is actually used. Patient memory Clinic memory Why the separation matters Patient memory is about continuity. A returning patient can have prior visit context, previous actions, and outcomes that are relevant to the current consultation. The goal is to avoid forcing the clinician to reconstruct that history from scratch every time the patient returns. Clinic memory answers a different question: what can we learn from the clinic as a whole? A useful pattern from one case may be relevant to another case with similar characteristics. The system therefore stores clinic-level experience in an anonymised form, keeping only an age band, sex, the problem, the treatment and the result, so it can contribute to future reasoning without being presented as another patient's chart. If every memory were stored in a single undifferentiated bank, the product would have a harder time explaining the provenance and scope of retrieved context. A clinician needs to know whether a remembered fact is about this patient or whether it reflects a broader clinic pattern. Separating the scopes makes the UI clearer and the recall logic easier to reason about. How Hindsight enforces the boundary Every patient memory is retained with a tag for that patient, and every patient recall asks for that tag with strict matching. Here is the core of the recall step. Patient recall is always scoped to one patient with strict tag matching, and it runs two queries in parallel: const scope = { tags: [patientTag(patient.id)], tagsMatch: "any_strict" as const }; Clinic Desk Content Pack · Page 8 of 24const [byComplaint, safety] = await Promise.all([ hs.recall(PATIENT_BANK, ${patient.name}: ${complaint}. Previous visits, treatments tried, and results., { ...scope, budget: "mid" }), hs.recall(PATIENT_BANK, ${patient.name} allergies, drug reactions, current medications, { ...scope, budget: "low" }), ]); One query follows today's complaint. The other always pulls allergies and current medicines, because safety facts must surface even when they have nothing to do with the reason for the visit. The any_strict match means an untagged memory, or another patient's memory, can never appear in this recall. Designing for transparency A memory system can feel mysterious if the UI only shows the final answer. We built a memory panel that explicitly shows what Hindsight recalled for the brief, grouped by patient and clinic, and lets the user explore the wider chart. That makes memory visible without turning the product into a developer console. Before and after The clearest way to see the value is one real consultation. A returning patient, 58, comes in with a dry cough that has lasted more than three months and asks for a stronger antibiotic. Without memory, the assistant only sees today's complaint. The brief says: likely viral or post-viral cough, consider antibiotics if there are signs of infection, cough syrup and steam. It sounds reasonable, and it is wrong for this patient. With Hindsight memory, the same complaint produces a very different brief. A penicillin allergy is on record, so amoxicillin is ruled out. The cough started about five weeks after lisinopril was prescribed for blood pressure. Azithromycin was already tried and made no difference, and a chest X-ray was normal. Across the clinic, several similar patients on ACE inhibitors resolved within weeks after switching to an ARB. Same model, same complaint, same prompt structure. The only thing that changed was what the agent remembered. Retaining outcomes The architecture also treats the outcome of an interaction as new information. Once a clinician records the result of a visit, Clinic Desk retains that learning twice: the full record into the patient scope, and a names tripped summary into the clinic scope. That creates a controlled feedback loop instead of an unbounded memory dump. Honest limitations Clinic Desk is decision support, not a doctor. It never gives a final diagnosis and never suggests doses. The data in the project is synthetic, and memory is only as good as what gets recorded: if a clinician skips the outcome step, the system has nothing new to learn from. Extraction also takes a moment after retain, so a memory saved seconds ago may not be recallable instantly. Lessons The broader engineering lesson is that persistent memory needs explicit boundaries, provenance, and a lifecycle. We found it easier to build the interface once those concepts were defined: recall has a source, Clinic Desk Content Pack · reasoning has context, the output has evidence, and the outcome becomes something that can be retained. Closing Clinic Desk uses Hindsight not simply to remember more, but to remember in the right place. Patient memory supports continuity. Clinic memory supports reusable experience. Together they give the reasoning layer more context while keeping the scope of each memory understandable to the person using the system. Links Hindsight GitHub: https://github.com/vectorize-io/hindsight Hindsight docs: https://hindsight.vectorize.io/ What is agent memory (Vectorize): https://vectorize.io/what-is-agent-memory Project repo: https://github.com/Abhinav1480/clinic_desk
Top comments (0)