I taught an incident agent to remember failed fixes

jeevan.m gowda — Sat, 07 Nov 2026 12:16:31 +0000

I taught an incident agent to remember failed fixes

At 3 a.m., “restart the pods” is a plausible suggestion right up until you remember that the last restart made the same incident worse. Most incident tools can show you what is happening now. I wanted an agent that could also tell the on-call engineer what the team had already tried, and what happened next.

That became RecallOps: an incident-response agent built around a simple distinction. Postgres stores operational facts; Hindsight stores experience. The agent uses both to investigate an alert, explain its hypotheses, and carry the result of a human decision into the next incident.

One incident, several kinds of context

The workflow starts when an alert arrives. The backend creates an incident record, then runs a bounded triage loop. It can look up recent deploys, logs, service ownership, and similar incidents. It streams events to the console as it works, including tool calls, memory hits, and a final suggestion. An engineer can accept, reject, or correct that suggestion, then resolve the incident with a root cause and fix.

Those sources serve different jobs. Postgres is the system of record for services, deploys, alerts, log lines, incident status, and feedback. Hindsight is where we keep compact accounts of what happened and the outcome of actions. Raw logs stay queryable in Postgres; putting every line into long-term memory would create noise and make the agent's evidence harder to inspect.

That separation also gives the interface something concrete to show. The Memory Panel can display the recalled text, its source incident ID and date, and whether a prior action worked, failed, or was partial. The engineer can see why an answer was suggested rather than being asked to trust a black box.

[INSERT SCREENSHOT HERE: RecallOps triage console showing the live agent stream beside the Memory Panel]

The most useful memory may be a warning

We first had to define what “remember an incident” meant. A useful memory is not a transcript. It is a small, dated, attributable record that connects a service and symptom to attempted actions, outcomes, cause, and resolution. For example, the project’s retain format captures a checkout latency incident, a failed pod restart, a connection-pool configuration change, and the rollback that resolved it.

The shape matters because the agent needs to retrieve evidence for a decision, not just find a document with similar words. A failure should remain attached to its context: which service, which signature, under what conditions, and how many times the action failed. Without those details, “restart failed” can be overgeneralized into an unsafe rule.

The workflow uses Hindsight’s retain, recall, and reflect capabilities. Retain records a postmortem or a fix outcome with a context tag and timestamp. Recall asks several narrower questions: what incidents resemble this symptom and service, what recent deploy pattern matches, and which fixes failed for this signature. Results are merged and deduplicated. Reflect is used to ask what recurring patterns show up across incidents, such as latency spikes that followed a pool configuration change.

Here is the documented retain/recall shape, simplified to the fields that carry the important semantics:

hs.retain(
    bank_id="kartly-sre",
    content="INC-1987 checkout-api latency. Restart failed; pool rollback fixed it.",
    context="incident_postmortem",
    timestamp="2026-09-22T03:14:00Z",
)

hits = hs.recall(
    bank_id="kartly-sre",
    query="checkout-api latency spike after deploy",
    budget="mid",
)

The timestamp is not decoration. “This happened before” is weaker than “this happened six weeks ago, after a similar deploy.” Temporal context helps an engineer judge whether an old memory is likely to apply. Hindsight’s documentation describes the memory operations; we used that model to make outcomes and time part of our incident records rather than treating recall as a generic nearest-neighbor search. For background on the broader idea, Vectorize’s explanation of agent memory is useful context.

A failed action must compete with a working one

Negative memory is not just a red badge in the UI. It changes the recommendation process. When the agent finds similar incidents, the candidate hypotheses are informed by similarity, recency, and outcome. A prior failed action should lower the appeal of repeating it; a successful fix under a similar signature should raise the value of investigating that path. The result is still a hypothesis for a human to assess, not an instruction to execute blindly.

The agent prompt is expected to cite incident IDs, state confidence, and warn about previously failed fixes. We also validate cited IDs against Postgres so a model cannot invent a source. That turns “I remember this” into a checkable claim: the engineer can open the incident and inspect the evidence.

Memory is optional in the triage path, too. The same alert and agent can run with Hindsight calls enabled or skipped. That is useful for two reasons: it makes the product’s behavior legible in a side-by-side comparison, and it gives evaluation a baseline. If the memory-enabled answer is not better on a held-out case, the design should reveal that rather than hiding it behind a polished demo.

Replay is a product test, not a chart generator

The project plan calls for a chronological replay harness. Earlier incidents form the history available to memory; later incidents are held out as replay cases. For each case, the harness records whether the correct root cause appears first or among the top three, whether a previously failed fix is repeated, and how long the simulated path takes to reach the right hypothesis. The memory-enabled run retains outcomes as it proceeds; the memory-off run answers without those Hindsight calls.

That setup matters more than a headline number. If the replay cases leak into the memory bank, the agent is looking at answers it has already seen. If the harness scrambles time, it can accidentally give the agent knowledge from the future. Keeping incidents in order and separating the held-out set makes the experiment easier to explain and reproduce.

for incident in replay_cases_in_time_order:
    answer = triage(incident, memory=memory_enabled)
    record_metrics(incident, answer)
    if memory_enabled:
        retain_outcome(incident, answer)

The learning curve is a way to inspect behavior across a sequence, not proof that every individual recommendation is safe. A useful evaluation should show the cases where memory helped, the cases where it did not, and whether the agent kept proposing an action that had failed before. The workflow describes a 40-incident replay and a separate history set; because the repository documents the intended evaluation but does not include final run output, I would publish measured values only after running the harness and checking that the replay bank contains no held-out cases.

[INSERT SCREENSHOT HERE: Memory-on versus memory-off replay chart with the evaluation split and metric labels visible]

What the engineer sees

Consider a checkout API alert for elevated p99 latency. With memory disabled, the agent can inspect current logs and recent deploys, but it has no record of how earlier incidents ended. It may offer a generic checklist, including restarting pods.

With memory enabled, it can surface an earlier checkout incident, identify the pool configuration change associated with it, and show that restarting pods failed while a rollback resolved the issue. It can also name the service owner from the operational database. The recommendation is more useful because it combines current system facts with a dated, outcome-bearing memory. The engineer still decides what to do.

The feedback loop closes the gap between a suggestion and what actually happened. If an engineer marks a fix as failed, RecallOps stores that outcome with its incident context. On resolution, it retains the postmortem. Later, reflect can summarize repeated patterns, and the Memory Panel can attribute those observations to supporting incidents. A correction such as “this one was Redis eviction, not the pool” is especially valuable: it prevents superficial symptom matching from becoming a confident but stale rule.

[INSERT SCREENSHOT HERE: Example incident detail with a failed-fix warning, supporting incident citation, and engineer feedback controls]

What I learned

Store outcomes, not just events. The fact that someone ran a command is less useful than whether it worked, failed, or made the incident worse.
Keep memory compact and sourced. Summaries are easier to retrieve and review than raw log dumps. Incident IDs and timestamps let humans verify claims.
Treat time as part of the evidence. Recency and temporal ordering affect whether a past resolution is relevant and whether an evaluation is honest.
Make the baseline visible. A memory-on/off path exposes the contribution of memory and gives the team a way to investigate when it does not help.
Let humans correct the record. Feedback is not merely UI polish. It is the new evidence the system needs to improve future triage.

Incident response is full of local knowledge: the rollback that worked, the restart that made things worse, and the engineer who knows why. RecallOps is an attempt to make that knowledge available at the moment it matters, with enough provenance that the on-call person can question it. The useful agent is not the one that always sounds certain. It is the one that can show what it remembers, where that memory came from, and how the team can correct it.

DEV Community: jeevan.m gowda

I taught an incident agent to remember failed fixes

I taught an incident agent to remember failed fixes

One incident, several kinds of context

The most useful memory may be a warning

A failed action must compete with a working one

Replay is a product test, not a chart generator

What the engineer sees

What I learned