Designing an AI incident agent for 3 a.m.: making memory visible

Nandish — Sat, 07 Nov 2026 17:08:12 +0000

Picture the person who uses an incident tool. It's the middle of the night. An alert just woke them up. Customers are affected, a manager is asking for updates, and they are trying to read logs with half-open eyes.

That person doesn't need a clever chatbot. They need an answer they can trust in seconds, and proof they can check.

Our team built DejaVu, an AI on-call agent that remembers how past outages were fixed. This post is about the design side: how we made an invisible thing, memory, visible and useful.

The engine underneath

DejaVu's memory is powered by Hindsight, an open-source agent memory system by Vectorize. Every past incident is stored in Hindsight. When a new error comes in, DejaVu recalls similar incidents, an LLM (GPT-OSS 120B on Groq) reasons over them, and the engineer's feedback is retained as a new memory.

If you're curious why memory is different from just searching documents, Vectorize explains it well in what is agent memory.

The challenge for design: memory works in the background. How do you show a stressed engineer why the agent is so sure?

1. A live memory map

The first thing you see is a map of dots. Each dot is a past incident, sized by how long it took to fix, and grouped by type of failure: cache, database, bad config, queues.

When DejaVu recalls incidents for a new error, the matching dots light up and lines draw out to them. You don't have to read anything to understand "this has happened before, many times." A failed fix shows as a red dot, and a newly learned incident appears as a green one.

2. Show the comparison, not just the answer

We added a button: Compare without vs with memory. It runs the same model twice, side by side. Without memory: 20% confident, a generic guess. With memory: 98%, with the exact fix.

This turned out to be the clearest way to explain the product to anyone, technical or not. One glance, and the value of memory is obvious.

3. Proof under every claim

The confidence ring never stands alone. Under it is a sentence in plain words, like "5 of 5 matching past incidents were fixed this way," and chips for each cited incident. On the right, each recalled incident has a card: the service, what happened, how strongly it matches, how long it took to fix, and why it matches.

The rule we followed: if the agent claims it, the screen shows where it came from.

4. One click to teach it

Feedback has to be effortless at 3 a.m. So it's two big buttons: Fix worked and Fix failed. One click saves the outcome to Hindsight in the background, and the interface never freezes:

learn_new_incident({
    "id": new_id,
    "service": ss.service,
    "error_signature": ss.error,
    "root_cause": answer["probable_cause"],
    "resolution_steps": answer["fix_steps"],
    "fix_successful": outcome,          # True or False from the button
    "time_to_resolve_minutes": minutes_since(ss.diag_at),
    "occurred_at": datetime.now(timezone.utc).isoformat(),
})

Inside, this calls Hindsight's retain with retain_async=True. A small celebration, a toast and a new green dot confirm it worked. We also record the real minutes from diagnosis to fix, not the AI's guess, so the learning curve stays honest.

5. Calm colours, red only for alerts

Our first version had neon colours everywhere, and it felt stressful. We switched to a calm black and champagne-gold theme. Red is used only for real alerts and failed fixes, so when you see red, it means something. Animations respect the "reduce motion" setting, and the layout works on a phone, because on-call engineers aren't always at a desk.

6. Fitting into a real team

Good design also means fitting how teams already work. Our path to adoption:

Import history: load past postmortems into Hindsight.
Alert in: a PagerDuty or Grafana webhook sends each alert to DejaVu.
Fix plan out: the cited diagnosis is posted to the team's Slack or Teams channel.
Close the loop: each "fix worked" becomes memory, and a weekly reflect report tells the team lead what keeps breaking.

Steps 1 and 4 work today. The Hindsight docs cover the reflect feature that powers that weekly report.

What I learned

Make the invisible visible. Memory is the core of the product, so we gave it a picture.
Proof builds trust faster than polish.
Design for the worst moment, not the demo moment.

Small details that made a big difference

A few smaller choices helped more than we expected:

Quick-pick alerts. Three buttons replay common outages (cache error after a deploy, database out of connections, login rate limits), so anyone can try the product without typing an error.
A downloadable postmortem. One click writes a blameless postmortem in Markdown, because writing it is the task everyone puts off after an outage.
A learning curve chart. It shows time-to-fix for every incident, with the average dropping from 44 minutes to 6 as memory grew. It answers "is this actually helping?" without a single word.
Friendly errors. When the free AI tier is busy, the app says "wait 30 seconds" instead of showing a stack trace.

None of these are about AI. They're about respecting the person using the tool.

Try it

Code: https://github.com/Pratheeksh-7S/dejavu-incident-agent
Demo video: https://youtu.be/Nu_sktguGMk
Hindsight: https://github.com/vectorize-io/hindsight

The best incident tool is the one a tired engineer can understand in five seconds.

DEV Community: Nandish