How We Built Clinic Desk Around Persistent Agent Memory

Vedant — Tue, 29 Sep 2026 16:09:15 +0000

Most AI assistants are good at answering the question in front of them. The harder problem is making the
next question benefit from what happened before. That was the design problem behind Clinic Desk.

What we built
Clinic Desk is a clinical intelligence workspace for a fictional family clinic. The interface combines a patient
queue, a consultation workspace, persistent memory, evidence-backed briefs, outcome recording, and an
insights view. The important part is not the number of screens. It is the memory loop connecting them.
Why memory had to be central
A consultation starts with a current reason for visit, but a returning patient may have history that changes
how the current situation should be interpreted. At the same time, a clinic accumulates experience across
many cases. We therefore separated two memory scopes: patient memory and anonymised clinic memory.
Patient memory keeps continuity for the same person; clinic memory captures reusable experience without
carrying patient-specific details into another case.
The recall step
When a clinician prepares a brief, Clinic Desk does not simply send the current complaint to the reasoning
model. The server first retrieves relevant memories from Hindsight. The recalled context can include
patient-specific facts as well as relevant clinic experience. This is the point where Hindsight becomes part
of the reasoning workflow rather than a storage box hidden behind the application.
Here is the core of the recall step. Patient recall is always scoped to one patient with strict tag matching,
and it runs two queries in parallel:
const scope = { tags: [patientTag(patient.id)], tagsMatch: "any_strict" as const };
const [byComplaint, safety] = await Promise.all([
hs.recall(PATIENT_BANK,
${patient.name}: ${complaint}. Previous visits, treatments tried, and results.,
{ ...scope, budget: "mid" }),
hs.recall(PATIENT_BANK,
${patient.name} allergies, drug reactions, current medications,
{ ...scope, budget: "low" }),
]);

One query follows today's complaint. The other always pulls allergies and current medicines, because
safety facts must surface even when they have nothing to do with the reason for the visit. The any_strict
match means an untagged memory, or another patient's memory, can never appear in this recall.
From recall to reasoning
The application then combines the retrieved context with the current visit and sends that grounded context
through the reasoning layer. The UI makes this visible: the clinician can see the brief, inspect supporting
citations, and understand which remembered items contributed to the answer. This was an important UX
decision because persistent memory is more useful when the user can see why it was used.
Comparison mode
We also added a comparison mode so the interface can contrast a memoryless response with a response
that includes recalled context. That creates a simple before-and-after experiment inside the product.
Instead of saying memory matters, the interface lets the user observe the behavioural difference for the
same consultation.
Before and after
The clearest way to see the value is one real consultation. A returning patient, 58, comes in with a dry
cough that has lasted more than three months and asks for a stronger antibiotic.
Without memory, the assistant only sees today's complaint. The brief says: likely viral or post-viral cough,
consider antibiotics if there are signs of infection, cough syrup and steam. It sounds reasonable, and it is
wrong for this patient.
With Hindsight memory, the same complaint produces a very different brief. A penicillin allergy is on record,
so amoxicillin is ruled out. The cough started about five weeks after lisinopril was prescribed for blood
pressure. Azithromycin was already tried and made no difference, and a chest X-ray was normal. Across
the clinic, several similar patients on ACE inhibitors resolved within weeks after switching to an ARB.
Same model, same complaint, same prompt structure. The only thing that changed was what the agent
remembered.

The other half: retain
Recall is only half of a learning system. After a consultation, the clinician records what happened. Clinic
Desk retains that outcome so it becomes part of the memory available to future relevant interactions. The
loop is therefore: recall, reason, act, record the outcome, retain, and recall again.
A practical engineering lesson
One of the biggest lessons from building Clinic Desk was that memory needs boundaries. We had to make
the distinction between patient memory and clinic memory explicit in the data model and in the interface.
That made both the product story and the safety model easier to understand.
Honest limitations
Clinic Desk is decision support, not a doctor. It never gives a final diagnosis and never suggests doses. The
data in the project is synthetic, and memory is only as good as what gets recorded: if a clinician skips the
outcome step, the system has nothing new to learn from. Extraction also takes a moment after retain, so a
memory saved seconds ago may not be recallable instantly.
What we learned

First, persistent memory should change behaviour, not just add another database table. Second, evidence
links matter when a generated brief depends on retrieved context. Third, outcome capture is what lets a
memory system improve over time. Finally, a memory feature is easiest to evaluate when the UI makes the
recall and learning steps visible.
Closing
Clinic Desk is our attempt to treat memory as a first-class part of an AI workflow. With Hindsight, the system
can recall relevant experience, use it during reasoning, capture outcomes, and make those outcomes
available to future interactions. That turns a one-shot assistant into a workflow that accumulates useful
context over time.

Links
Hindsight GitHub: https://github.com/vectorize-io/hindsight
Hindsight docs: https://hindsight.vectorize.io/
What is agent memory (Vectorize): https://vectorize.io/what-is-agent-memory
Project repo: https://github.com/Abhinav1480/clinic_desk

DEV Community: Vedant

How We Built Clinic Desk Around Persistent Agent Memory