Retain, recall, reflect: building an incident agent's memory with Hindsight

Our team built DejaVu, an AI agent that helps on-call engineers by remembering how past outages were fixed. The "remembering" part is done by Hindsight, an open-source agent memory system by Vectorize.

This post is a practical tour of how we used Hindsight's four main operations, with the real code, plus the small decisions that made the biggest difference.

If you want the big-picture idea first, read Vectorize's explainer on what agent memory is.

Running Hindsight locally

We ran the Hindsight server on our own laptops (pip install hindsight-api), with Groq as its LLM provider. That made the whole stack free to run. The Python client talks to it at http://localhost:8888.

1. create_bank: give memory a purpose

A memory bank is a separate store of memories. We made one called dejavu-incidents and gave it a mission, which tells Hindsight what is worth remembering:

client.create_bank(
    bank_id="dejavu-incidents",
    name="DejaVu incident memory",
    mission=(
        "Remember every production incident: the error signature, the root cause, "
        "the exact fix and whether it worked, so future outages are solved faster."
    ),
)

The mission sounds like a small thing, but it guides which facts Hindsight pulls out of each incident.

2. retain: store each incident as plain English

Every past incident goes in with retain. Three details made this work well:

client.retain(
    bank_id="dejavu-incidents",
    content=incident_to_text(inc),            # plain-English summary
    context="past production incident and how it was resolved",
    timestamp=inc["occurred_at"],             # when it really happened
    document_id=f"incident-{inc['id']}",      # stable id
    metadata={"incident_id": str(inc["id"]), "service": inc["service"]},
    retain_async=background,                  # True for live learning
)

• Plain English beats raw JSON. We turn each incident into sentences ("On payments-api, the error was... The root cause was... The fix worked."). Hindsight extracts much better facts from text a human would write.
• A stable document_id. Running the loader twice updates incident 12 instead of creating a duplicate.
• Metadata. Storing incident_id on every fact lets us group recalled facts back into incidents later.

When an engineer clicks Fix worked in the app, we retain the new incident with retain_async=True. It's saved in the background, so the UI never freezes.

3. recall: find similar incidents, even with different words

When a new error arrives, recall returns the facts most related to it. Hindsight combines several search strategies and reranks the results, so it finds matches on meaning, not just keywords. In our data, one cache bug appeared with 6 different error messages on 3 services, and recall found them all.

Our key decision was how to rank. Hindsight gives each fact several scores. We rank by the reranker score, not recency, so an old but identical outage still comes first:

for fact in response.results:
    incident_id = int(fact.metadata["incident_id"])
    score = fact.scores.reranker if fact.scores.reranker is not None else fact.scores.final
    entry = matches.setdefault(incident_id, {"match": 0.0, "facts": []})
    entry["match"] = max(entry["match"], score)
    entry["facts"].append(fact.text)

Our first version leaned on recency and kept missing a perfect match from months earlier. Switching to the reranker score fixed it.

4. reflect: ask questions across all memories

recall answers "what is similar to this?". reflect answers bigger questions by reasoning over the whole bank. We added a Lessons from memory panel with three buttons for a team lead:

• What keeps breaking?
• Which fixes failed, and why?
• What should we fix for good?

Reflect makes several LLM calls, so on Groq's free tier (about 8,000 tokens a minute) we keep it light with a low budget and token caps. If it's over the limit, the app falls back to recall plus one LLM call, and the card says which one answered. The Hindsight docs cover reflect's options in detail.

Lessons from building it

1. Write memories for a reader. Plain sentences produced better facts than structured dumps.
2. Keep ids stable so re-runs update instead of duplicating.
3. Use metadata to map facts back to your own objects.
4. Choose your ranking on purpose. Relevance or recency is a product decision, not a default.
5. Plan for rate limits. Our loader waits 40 seconds between incidents, and the app has a fallback.

How the pieces fit together

Here's the full loop for one outage, in order:

1. The engineer pastes an error into the Streamlit dashboard.
2. recall returns the most relevant facts, and we group them into the top 5 incidents.
3. We send only those incidents to GPT-OSS 120B, with strict instructions to cite only what memory returned.
4. The answer comes back with a root cause, fix steps and evidence. We clean the evidence and calibrate the confidence from proven fixes.
5. The engineer clicks Fix worked or Fix failed, and retain stores the new incident in the background.
6. Later, reflect uses everything, including that new incident, to answer the team lead's questions.

Every step either reads from memory or writes to it. That's why we call Hindsight the core of the project rather than a feature.

Try it

• Code: https://github.com/Pratheeksh-7S/dejavu-incident-agent
• Demo video: https://youtu.be/Nu_sktguGMk
• Hindsight: https://github.com/vectorize-io/hindsight
• Docs: https://hindsight.vectorize.io

If you're building any agent that should get better over time, these four calls are a great place to start.