RecallOps: AI Incident Response Copilot

Balaji Chowdary Puvvada — Mon, 28 Sep 2026 11:59:46 +0000

Hindsight-Powered Incident Investigation

Incident response is a race against missing context. When an alert fires, an on-call engineer usually has plenty of live data — dashboards, logs, error rates — but almost never has a quick way to ask "has this happened before, and how was it fixed?" That gap is what our team set out to close with RecallOps, an AI incident-response copilot. I worked on the frontend and the investigation experience, so this article looks at the project from that angle: how the interface turns a wall of alerts into a workspace where past incidents show up exactly when they're useful.

Designing for the moment of confusion

The instinct when something breaks is to start digging through metrics and logs from scratch, even if a teammate solved the same problem months ago. An AI assistant without memory runs into the same wall — it can read the current symptoms but has no sense of your system's history. The interface I worked on had to solve for that specific moment: the few minutes right after an incident opens, when an engineer needs relevant history without having to go hunting for it themselves.

RecallOps is organized around seven areas — Dashboard, Incidents, Incident Workspace, Copilot, Memory, Learning, and Services — and the Incident Workspace is where the investigation actually happens. From that screen, an engineer can look at the current incident, trigger AI analysis, pull up historical memory, review why a past incident was judged similar, compare current and historical evidence side by side, work through structured investigation paths, and resolve the incident.

Screenshot suggestion 1: A full view of the Incident Workspace for INC-017, showing the layout — current incident summary, AI investigation console, and the operational memory panel all visible at once. Caption: "The Incident Workspace lays out live evidence, AI analysis, and recalled memory in one screen."

Walking through INC-017

The clearest example of this in action is the INC-001 / INC-017 pair from the demo data. INC-001 was a Payment API database timeout caused by connection-pool exhaustion from a connection leak; it was resolved by fixing the leak and increasing pool capacity, then retained as memory. When INC-017 later came in — another Payment API database timeout — the workspace didn't just show current metrics. It surfaced INC-001 automatically as the top historical match, at 91% similarity.

That number isn't just decoration. It sits next to a "Why this memory?" list that explains the match in plain terms: same service, same service family, similar symptoms, similar database behavior, similar timing. As someone focused on the investigation UI, this list mattered more to me than the percentage itself — a similarity score alone tells you "trust this," but the reasons let the engineer actually evaluate the match instead of taking it on faith.

Screenshot suggestion 2: A close-up of the Operational Memory card for INC-017, showing the 91% score and the "Why this memory?" checklist. Caption: "INC-001 recalled at 91% similarity, with the specific reasons behind the match."

Keeping current and historical evidence separate

One decision that shaped a lot of the frontend work was refusing to blend live data with historical data into a single summary. The workspace keeps four things visibly distinct: current evidence, historical evidence, the AI recommendation, and uncertainty.

For INC-017, current evidence included 98% connection utilization, a 14.2% timeout rate, a deployment 23 minutes earlier, and a connection-timeout error family. Historical evidence, pulled from INC-001, showed 96% connection utilization, the same service and error family, and a confirmed connection leak as the root cause. Laying these out in two clearly labeled panels, rather than one merged narrative, was a deliberate choice — it's much easier to sanity-check a recommendation when you can see exactly which numbers came from now and which came from before.

Screenshot suggestion 3: The side-by-side "Current Evidence" and "Historical Evidence" panels for INC-017. Caption: "Live metrics on the left, INC-001's historical data on the right — nothing is merged into a single blob."

Recommendation, uncertainty, and why that pairing matters

The overlap between INC-017 and INC-001 is genuinely useful — same service, same error family, matching utilization numbers — but it isn't proof that the two incidents share a root cause. RecallOps treats the historical match as evidence to investigate, not a conclusion to accept, and the UI keeps uncertainty visible right next to the AI's recommendation. The workspace also offers structured investigation paths, so an engineer can go check the hypothesis directly — for example, whether the recent deployment introduced a new leak — instead of just trusting the pattern match.

From a design standpoint, this was the hardest part to get right. It's tempting to make a 91% match feel like an answer. But an interface that oversells confidence teaches engineers to stop checking their own work, which is the opposite of what you want during an incident. Showing the uncertainty alongside the recommendation was as important as showing the recommendation itself.

No automatic changes to production

This is worth stating plainly because it shaped every screen I worked on: RecallOps never touches production systems on its own. It surfaces evidence, history, recommendations, investigation paths, and uncertainty — the engineer makes the final call. That constraint pushed the interface toward being a research and decision-support tool rather than an automation tool, which is also why so much of the layout is built around comparison (current vs. historical) rather than around a single "do this" button.

Retain and reflect

Once an engineer resolves INC-017 and retains it, it becomes part of memory for future recall. Beyond individual incidents, the Learning area looks across retained incidents for recurring patterns — on the demo data, it surfaces a recurring Payment API pattern spanning five related incidents, along with the evidence trail behind each synthesized lesson, so an engineer can trace a "lesson" back to the specific incidents it came from.

What I took away from this

A few things stuck with me from working on this part of the product:

Separating evidence sources is a UI problem, not just a data problem. It's not enough for the backend to keep current and historical data distinct — the interface has to keep reinforcing that separation, or engineers will start treating a recalled memory as confirmed fact.
A similarity score needs a reason attached. A bare percentage invites blind trust; a "why this memory?" breakdown invites verification.
Uncertainty is a feature, not a gap. Showing what the system doesn't know is what makes the parts it does show trustworthy.

RecallOps is still a demo-stage project — the verified path runs on SQLite with a deterministic AI fallback, while a production Postgres deployment, a live Hindsight service, and live LLM inference are configured but not yet tested end-to-end. But the core interaction — recall a similar past incident, show why it matched, compare it against what's happening now, and let the engineer decide — is the piece I'm most interested in continuing to refine.

Screenshot suggestion 4: The Learning area showing the recurring Payment API pattern across five incidents. Caption: "The Learning view traces a recurring pattern back to the incidents that produced it."

DEV Community: Balaji Chowdary Puvvada

RecallOps: AI Incident Response Copilot