DEV Community: kushal0685

Same AI, two answers: what happened when we gave our on-call agent a memory.

kushal0685 — Sat, 07 Nov 2026 18:03:59 +0000

Picture the person who uses an incident tool. It's the middle of the night. An alert just woke them up. Customers are affected, a manager is asking for updates, and they are trying to read logs with half-open eyes.

That person doesn't need a clever chatbot. They need an answer they can trust in seconds, and proof they can check.

Our team built DejaVu, an AI on-call agent that remembers how past outages were fixed. This post is about the design side: how we made an invisible thing, memory, visible and useful.

The engine underneath

DejaVu's memory is powered by Hindsight, an open-source agent memory system by Vectorize. Every past incident is stored in Hindsight. When a new error comes in, DejaVu recalls similar incidents, an LLM (GPT-OSS 120B on Groq) reasons over them, and the engineer's feedback is retained as a new memory.

If you're curious why memory is different from just searching documents, Vectorize explains it well in what is agent memory.

The challenge for design: memory works in the background. How do you show a stressed engineer why the agent is so sure?

1. A live memory map

The first thing you see is a map of dots. Each dot is a past incident, sized by how long it took to fix, and grouped by type of failure: cache, database, bad config, queues.

When DejaVu recalls incidents for a new error, the matching dots light up and lines draw out to them. You don't have to read anything to understand "this has happened before, many times." A failed fix shows as a red dot, and a newly learned incident appears as a green one.

2. Show the comparison, not just the answer

We added a button: Compare without vs with memory. It runs the same model twice, side by side. Without memory: 20% confident, a generic guess. With memory: 98%, with the exact fix.

This turned out to be the clearest way to explain the product to anyone, technical or not. One glance, and the value of memory is obvious.

3. Proof under every claim

The confidence ring never stands alone. Under it is a sentence in plain words, like "5 of 5 matching past incidents were fixed this way," and chips for each cited incident. On the right, each recalled incident has a card: the service, what happened, how strongly it matches, how long it took to fix, and why it matches.

The rule we followed: if the agent claims it, the screen shows where it came from.

4. One click to teach it

Feedback has to be effortless at 3 a.m. So it's two big buttons: Fix worked and Fix failed. One click saves the outcome to Hindsight in the background, and the interface never freezes:

learn_new_incident({
    "id": new_id,
    "service": ss.service,
    "error_signature": ss.error,
    "root_cause": answer["probable_cause"],
    "resolution_steps": answer["fix_steps"],
    "fix_successful": outcome,          # True or False from the button
    "time_to_resolve_minutes": minutes_since(ss.diag_at),
    "occurred_at": datetime.now(timezone.utc).isoformat(),
})

Inside, this calls Hindsight's retain with retain_async=True. A small celebration, a toast and a new green dot confirm it worked. We also record the real minutes from diagnosis to fix, not the AI's guess, so the learning curve stays honest.

5. Calm colours, red only for alerts

Our first version had neon colours everywhere, and it felt stressful. We switched to a calm black and champagne-gold theme. Red is used only for real alerts and failed fixes, so when you see red, it means something. Animations respect the "reduce motion" setting, and the layout works on a phone, because on-call engineers aren't always at a desk.

6. Fitting into a real team

Good design also means fitting how teams already work. Our path to adoption:

Import history: load past postmortems into Hindsight.
Alert in: a PagerDuty or Grafana webhook sends each alert to DejaVu.
Fix plan out: the cited diagnosis is posted to the team's Slack or Teams channel.
Close the loop: each "fix worked" becomes memory, and a weekly reflect report tells the team lead what keeps breaking.

Steps 1 and 4 work today. The Hindsight docs cover the reflect feature that powers that weekly report.

What I learned

Make the invisible visible. Memory is the core of the product, so we gave it a picture.
Proof builds trust faster than polish.
Design for the worst moment, not the demo moment.

Small details that made a big difference

A few smaller choices helped more than we expected:

Quick-pick alerts. Three buttons replay common outages (cache error after a deploy, database out of connections, login rate limits), so anyone can try the product without typing an error.
A downloadable postmortem. One click writes a blameless postmortem in Markdown, because writing it is the task everyone puts off after an outage.
A learning curve chart. It shows time-to-fix for every incident, with the average dropping from 44 minutes to 6 as memory grew. It answers "is this actually helping?" without a single word.
Friendly errors. When the free AI tier is busy, the app says "wait 30 seconds" instead of showing a stack trace.

None of these are about AI. They're about respecting the person using the tool.

Try it

Code: https://github.com/Pratheeksh-7S/dejavu-incident-agent
Demo video: https://youtu.be/Nu_sktguGMk
Hindsight: https://github.com/vectorize-io/hindsight

The best incident tool is the one a tired engineer can understand in five seconds.

Same AI, two answers: what happened when we gave our on-call agent a memory

kushal0685 — Sat, 07 Nov 2026 17:59:05 +0000

That person doesn't need a clever chatbot. They need an answer they can trust in seconds, and proof they can check.

Our team built DejaVu, an AI on-call agent that remembers how past outages were fixed. This post is about the design side: how we made an invisible thing, memory, visible and useful.

The engine underneath

If you're curious why memory is different from just searching documents, Vectorize explains it well in what is agent memory.

The challenge for design: memory works in the background. How do you show a stressed engineer why the agent is so sure?

1. A live memory map

The first thing you see is a map of dots. Each dot is a past incident, sized by how long it took to fix, and grouped by type of failure: cache, database, bad config, queues.

2. Show the comparison, not just the answer

We added a button: Compare without vs with memory. It runs the same model twice, side by side. Without memory: 20% confident, a generic guess. With memory: 98%, with the exact fix.

This turned out to be the clearest way to explain the product to anyone, technical or not. One glance, and the value of memory is obvious.

3. Proof under every claim

The rule we followed: if the agent claims it, the screen shows where it came from.

4. One click to teach it

Feedback has to be effortless at 3 a.m. So it's two big buttons: Fix worked and Fix failed. One click saves the outcome to Hindsight in the background, and the interface never freezes:

learn_new_incident({
    "id": new_id,
    "service": ss.service,
    "error_signature": ss.error,
    "root_cause": answer["probable_cause"],
    "resolution_steps": answer["fix_steps"],
    "fix_successful": outcome,          # True or False from the button
    "time_to_resolve_minutes": minutes_since(ss.diag_at),
    "occurred_at": datetime.now(timezone.utc).isoformat(),
})

5. Calm colours, red only for alerts

6. Fitting into a real team

Good design also means fitting how teams already work. Our path to adoption:

Import history: load past postmortems into Hindsight.
Alert in: a PagerDuty or Grafana webhook sends each alert to DejaVu.
Fix plan out: the cited diagnosis is posted to the team's Slack or Teams channel.
Close the loop: each "fix worked" becomes memory, and a weekly reflect report tells the team lead what keeps breaking.

Steps 1 and 4 work today. The Hindsight docs cover the reflect feature that powers that weekly report.

What I learned

Make the invisible visible. Memory is the core of the product, so we gave it a picture.
Proof builds trust faster than polish.
Design for the worst moment, not the demo moment.

Small details that made a big difference

A few smaller choices helped more than we expected:

Quick-pick alerts. Three buttons replay common outages (cache error after a deploy, database out of connections, login rate limits), so anyone can try the product without typing an error.
A downloadable postmortem. One click writes a blameless postmortem in Markdown, because writing it is the task everyone puts off after an outage.
A learning curve chart. It shows time-to-fix for every incident, with the average dropping from 44 minutes to 6 as memory grew. It answers "is this actually helping?" without a single word.
Friendly errors. When the free AI tier is busy, the app says "wait 30 seconds" instead of showing a stack trace.

None of these are about AI. They're about respecting the person using the tool.

Try it

Code: https://github.com/Pratheeksh-7S/dejavu-incident-agent
Demo video: https://youtu.be/Nu_sktguGMk
Hindsight: https://github.com/vectorize-io/hindsight

The best incident tool is the one a tired engineer can understand in five seconds.

Retain, recall, reflect: building an incident agent's memory with Hindsight

kushal0685 — Sat, 07 Nov 2026 16:38:10 +0000

Our team built DejaVu, an AI agent that helps on-call engineers by remembering how past outages were fixed. The "remembering" part is done by Hindsight, an open-source agent memory system by Vectorize.

This post is a practical tour of how we used Hindsight's four main operations, with the real code, plus the small decisions that made the biggest difference.

If you want the big-picture idea first, read Vectorize's explainer on what agent memory is.

Running Hindsight locally

We ran the Hindsight server on our own laptops (pip install hindsight-api), with Groq as its LLM provider. That made the whole stack free to run. The Python client talks to it at http://localhost:8888.

1. create_bank: give memory a purpose

A memory bank is a separate store of memories. We made one called dejavu-incidents and gave it a mission, which tells Hindsight what is worth remembering:

client.create_bank(
    bank_id="dejavu-incidents",
    name="DejaVu incident memory",
    mission=(
        "Remember every production incident: the error signature, the root cause, "
        "the exact fix and whether it worked, so future outages are solved faster."
    ),
)

The mission sounds like a small thing, but it guides which facts Hindsight pulls out of each incident.

2. retain: store each incident as plain English

Every past incident goes in with retain. Three details made this work well:

client.retain(
    bank_id="dejavu-incidents",
    content=incident_to_text(inc),            # plain-English summary
    context="past production incident and how it was resolved",
    timestamp=inc["occurred_at"],             # when it really happened
    document_id=f"incident-{inc['id']}",      # stable id
    metadata={"incident_id": str(inc["id"]), "service": inc["service"]},
    retain_async=background,                  # True for live learning
)

Plain English beats raw JSON. We turn each incident into sentences ("On payments-api, the error was... The root cause was... The fix worked."). Hindsight extracts much better facts from text a human would write.
A stable document_id. Running the loader twice updates incident 12 instead of creating a duplicate.
Metadata. Storing incident_id on every fact lets us group recalled facts back into incidents later.

When an engineer clicks Fix worked in the app, we retain the new incident with retain_async=True. It's saved in the background, so the UI never freezes.

3. recall: find similar incidents, even with different words

When a new error arrives, recall returns the facts most related to it. Hindsight combines several search strategies and reranks the results, so it finds matches on meaning, not just keywords. In our data, one cache bug appeared with 6 different error messages on 3 services, and recall found them all.

Our key decision was how to rank. Hindsight gives each fact several scores. We rank by the reranker score, not recency, so an old but identical outage still comes first:

for fact in response.results:
    incident_id = int(fact.metadata["incident_id"])
    score = fact.scores.reranker if fact.scores.reranker is not None else fact.scores.final
    entry = matches.setdefault(incident_id, {"match": 0.0, "facts": []})
    entry["match"] = max(entry["match"], score)
    entry["facts"].append(fact.text)

Our first version leaned on recency and kept missing a perfect match from months earlier. Switching to the reranker score fixed it.

4. reflect: ask questions across all memories

recall answers "what is similar to this?". reflect answers bigger questions by reasoning over the whole bank. We added a Lessons from memory panel with three buttons for a team lead:

What keeps breaking?
Which fixes failed, and why?
What should we fix for good?

Reflect makes several LLM calls, so on Groq's free tier (about 8,000 tokens a minute) we keep it light with a low budget and token caps. If it's over the limit, the app falls back to recall plus one LLM call, and the card says which one answered. The Hindsight docs cover reflect's options in detail.

Lessons from building it

Write memories for a reader. Plain sentences produced better facts than structured dumps.
Keep ids stable so re-runs update instead of duplicating.
Use metadata to map facts back to your own objects.
Choose your ranking on purpose. Relevance or recency is a product decision, not a default.
Plan for rate limits. Our loader waits 40 seconds between incidents, and the app has a fallback.

How the pieces fit together

Here's the full loop for one outage, in order:

The engineer pastes an error into the Streamlit dashboard.
recall returns the most relevant facts, and we group them into the top 5 incidents.
We send only those incidents to GPT-OSS 120B, with strict instructions to cite only what memory returned.
The answer comes back with a root cause, fix steps and evidence. We clean the evidence and calibrate the confidence from proven fixes.
The engineer clicks Fix worked or Fix failed, and retain stores the new incident in the background.
Later, reflect uses everything, including that new incident, to answer the team lead's questions.

Every step either reads from memory or writes to it. That's why we call Hindsight the core of the project rather than a feature.

Try it

Code: https://github.com/Pratheeksh-7S/dejavu-incident-agent
Demo video: https://youtu.be/Nu_sktguGMk
Hindsight: https://github.com/vectorize-io/hindsight
Docs: https://hindsight.vectorize.io

If you're building any agent that should get better over time, these four calls are a great place to start.