A system that only recalls old information is useful. A system that can also learn from what happened next
is more interesting. That became the central loop behind Clinic Desk.
The workflow
Clinic Desk starts with a current consultation. The clinician sees the patient, the reason for the visit, and a
memory panel powered by Hindsight. When the clinician prepares a brief, the application recalls relevant
patient and clinic context before generating the response. This creates a clear path from current input to
remembered context to generated output.
Here is the core of the recall step. Patient recall is always scoped to one patient with strict tag matching,
and it runs two queries in parallel:
const scope = { tags: [patientTag(patient.id)], tagsMatch: "any_strict" as const };
const [byComplaint, safety] = await Promise.all([
hs.recall(PATIENT_BANK,
${patient.name}: ${complaint}. Previous visits, treatments tried, and results.,
{ ...scope, budget: "mid" }),
hs.recall(PATIENT_BANK,
${patient.name} allergies, drug reactions, current medications,
{ ...scope, budget: "low" }),
]);
One query follows today's complaint. The other always pulls allergies and current medicines, because
safety facts must surface even when they have nothing to do with the reason for the visit. The any_strict
match means an untagged memory, or another patient's memory, can never appear in this recall.
Why recall alone is incomplete
Suppose an assistant recalls that a certain approach was tried previously. That information is only part of
the story. The system also needs to know what happened after the action. Did the case improve? Was
there no change? Did the situation get worse? Recording the outcome gives the memory system a more
complete picture of experience.
The retain step
Clinic Desk therefore includes an outcome step after the consultation. The clinician marks the result as
resolved, improved, no change or worse, and saves it so the experience becomes part of persistent
memory. It is written into the patient's own chart and, with the name removed, into clinic memory. This is the
retain side of the Hindsight story, and it is the part that turns memory from a read-only feature into a
learning loop.
The loop
The architecture is easiest to explain as five steps: recall relevant experience, reason over that context, act
in the workflow, record the outcome, and retain the new information. On a future visit, the system can recall
what was retained. The important point is that the output of one interaction becomes input to another.
Before and after
The clearest way to see the value is one real consultation. A returning patient, 58, comes in with a dry
cough that has lasted more than three months and asks for a stronger antibiotic.
Without memory, the assistant only sees today's complaint. The brief says: likely viral or post-viral cough,
consider antibiotics if there are signs of infection, cough syrup and steam. It sounds reasonable, and it is
wrong for this patient.
With Hindsight memory, the same complaint produces a very different brief. A penicillin allergy is on record,
so amoxicillin is ruled out. The cough started about five weeks after lisinopril was prescribed for blood
pressure. Azithromycin was already tried and made no difference, and a chest X-ray was normal. Across
the clinic, several similar patients on ACE inhibitors resolved within weeks after switching to an ARB.
Same model, same complaint, same prompt structure. The only thing that changed was what the agent
remembered.
Making learning visible
We wanted the interface to expose this lifecycle. The Memory screen shows the stored context. The
consultation screen shows what was recalled for the current brief. The Insights screen gives the clinic a
broader view of accumulated memory and outcomes. These screens are different views of the same
memory lifecycle.
Evidence matters
When memory contributes to a generated brief, provenance becomes important. Clinic Desk uses citations
so that the clinician can inspect supporting memory rather than accepting the generated wording on trust.
This was especially useful while testing the application because we could see whether the retrieved context
was actually reflected in the output.
Honest limitations
Clinic Desk is decision support, not a doctor. It never gives a final diagnosis and never suggests doses. The
data in the project is synthetic, and memory is only as good as what gets recorded: if a clinician skips the
outcome step, the system has nothing new to learn from. Extraction also takes a moment after retain, so a
memory saved seconds ago may not be recallable instantly.
Engineering lessons
The biggest lesson was that learning needs a real event boundary. We needed a moment when the human
says, in effect, this is what happened. That makes retention meaningful and auditable. Another lesson was
to keep the memory scopes explicit so that patient continuity and clinic learning do not blur together.
Closing
Clinic Desk is built around the idea that an AI workflow should not forget immediately after producing an
answer. Hindsight gives us the primitives to recall relevant context and retain new experience. The
application connects those primitives into a visible workflow where the result of one interaction can shape
the context of a later one.
Links
Hindsight GitHub: https://github.com/vectorize-io/hindsight
Hindsight docs: https://hindsight.vectorize.io/
What is agent memory (Vectorize): https://vectorize.io/what-is-agent-memory
Project repo: https://github.com/Abhinav1480/clinic_desk
Top comments (0)