Why my agent refuses to repeat a fix that only helped briefly
At 2 a.m. the payments API ran out of database connections for the fourth time, and the first thing the on-call engineer did was restart it. It worked. For about eleven minutes.
Every incident tool I had used would have recorded that restart as "the fix". I wanted an agent that remembers the restart as what it really was: a lie that bought us time. That one requirement ended up shaping most of the system I built.
What the system does
Aftermath is an incident response agent for on-call engineers. An alert comes in, the agent recalls similar past incidents (the root cause, the fix that worked, and the fixes that did not), and it proposes a diagnosis and a plan. When the incident is resolved, the engineer's write-up is saved back into memory, so the next incident starts from a better place.
I own the backend: a FastAPI service with SQLite, a Server-Sent Events stream for the analysis, and a thin AgentService contract that keeps the HTTP layer and the reasoning engine apart. The agent only ever sees plain Pydantic models, never database rows:
class AgentService(Protocol):
def analyze(self, incident: IncidentContext) -> AsyncIterator[AgentEvent]
async def learn_from_resolution(self, incident: IncidentContext, resolution: ResolutionContext) -> None
async def ingest_history(self, incidents: list[IncidentContext]) -> None
analyze streams recalled memories first, then the explanation token by token, then one structured suggestion. learn_from_resolution is where memory gets written. That second method is where this article is going.
The memory seam
The reasoning engine talks to memory through a two-method protocol. Nothing else in the engine knows or cares what stores the records:
@runtime_checkable
class IncidentMemory(Protocol):
backend: str
async def recall(self, query: MemoryQuery) -> list[RecalledIncident]: ...
async def retain(self, record: MemoryRecord) -> None: ...
This is the shape of Hindsight's agent memory model: you retain what happened and you recall what is relevant. I deliberately made it the only way the engine touches history, because I wanted the memory layer to be swappable and honest about what it is. Every report carries a memory_backend field, so an analysis can never imply it consulted Hindsight when it actually used a local file.
The part worth copying is what goes into retain. A MemoryRecord renders itself as plain English, because a text-based memory system retains and recalls sentences far better than it handles a bag of database columns:
def retain_text(self) -> str:
lines = [f"Incident {self.incident_id} on {self.service}: {self.title}."]
if self.root_cause:
lines.append(f"Confirmed root cause: {self.root_cause}.")
if self.resolution:
lines.append(f"Fix that worked ({self.resolution_status.value}): {self.resolution}.")
for attempt in self.attempts:
if attempt.outcome != ResolutionStatus.VERIFIED:
lines.append(f"Attempt that did NOT resolve it ({attempt.outcome.value}): {attempt.description}.")
...
Notice the line about attempts that did not resolve it. Most incident memory stores the happy path. I store the dead ends with equal weight, because the dead ends are what stop the next engineer from wasting an hour.
The core story: a restart is not a fix
When an engineer resolves an incident they fill in three things: the root cause, the fix that worked, and what failed. Free text, written at the end of a bad night. The tempting design is to save it as typed and move on.
The problem is that people honestly write "restarted the service, it recovered" and mean it. If I save that as a verified fix, the agent will cheerfully recommend a restart for the next payments outage, and the next engineer will lose another eleven minutes before the pool fills up again.
So resolution text is classified before it is trusted. Each sentence maps to an action family (restart, scale out, raise a limit, rollback, code fix, and so on), and a second check looks for wording that signals relief rather than repair:
_TEMPORARY_HINTS = re.compile(
r"came back|returned|again within|again after|recurred|temporar|briefly|for (?:a few|\d+) min|bought|"
r"exhausted again|re-?occurred|only helped|short[- ]lived", re.I)
def classify_failed_attempt(text: str) -> ResolutionStatus:
"""A reported 'what failed' entry is either a pure failure or a temporary relief."""
return ResolutionStatus.TEMPORARY if _TEMPORARY_HINTS.search(text) else ResolutionStatus.FAILED
On top of that, when the engineer's own "fix that worked" is only a restart, scale-out, limit bump, failover, cache purge or wait, and nothing in the text says fix, patch or revert, it is recorded as temporary. It is never promoted to a verified root-cause fix. The rule in the module docstring is blunt: a failed attempt is never recorded as a solution, and a temporary mitigation is never recorded as a verified fix.
The rules are regexes and I know it. They are cheap, readable, and easy to argue with in a code review. I would rather debate a visible pattern than a hidden model.
Turning memory into warnings
Storing the dead ends only matters if recall changes what the agent says. After recall, the engine derives lessons from the matched incidents:
for action, entries in failed_by_action.items():
ids = tuple(dict.fromkeys(i for i, _ in entries))
outcomes = {a.outcome for _, a in entries}
kind = "only gave temporary relief" if outcomes == {ResolutionStatus.TEMPORARY} else "did not resolve it"
message = f"{label} {kind} in {', '.join(ids)} (e.g. \"{example[:140]}\")"
lessons.append(Lesson(kind="avoid_action", action_type=action, incident_ids=ids, message=message))
The remediation planner then uses those avoid_action lessons. A step whose action family already failed, or only gave temporary relief, in similar past incidents is moved out of the plan and into the warnings. State-changing steps are all marked as needing human approval, and nothing the model writes is ever executed.
Here is the before and after. With no memory, a payments-api pool exhaustion alert gets the standard playbook, which includes "restart the service" near the top. With memory holding the earlier incidents, the same alert comes back with the restart demoted to a warning that reads, in effect, "Restarting only gave temporary relief in INC-1001 and INC-1009", and the suggested fix points at the leaking connection in the origin frame of the stack trace.
Memory that can disagree with itself
Two incidents can look identical and have different causes. If the recalled incidents have similar symptoms but disjoint root-cause signatures, the engine does not pick a winner. It emits a contradiction lesson: similar symptoms had different root causes before, so confirm which one applies before reusing a past fix. A confident wrong answer is worse than a hedged right one at 2 a.m.
Recall itself is scored with weights I can explain: signatures 45%, service 20%, keywords 20%, stack and exception match 15%. A small recency bonus, at most 0.05 and gone after 180 days, is added after the threshold. It reorders good matches but never pulls in a new one, so a recent fix, which reflects the current code, wins a tie, while a clearly better old match still beats a weak recent one.
Memory is an attack surface
Resolution text is typed by humans and quoted back into later analyses and prompts. That makes memory a place where a bad string can live for months. If a past fix contains curl … | sh, sudo, rm -rf, $(…), or text like "ignore previous instructions", the engine keeps it in the record (it is what happened) but never repeats it as a plan step, never quotes it in a warning, and never sends it to the model. The plan tells you to read the original incident instead. Secrets are redacted before storage and before any model call, by the same function in both places.
What I measured, and what I did not
The seed history is twenty incidents across six services, with recurring patterns. The payments-api pool exhaustion appears three times, with a restart tried first each time and not helping. I generate that data deterministically, so I treat the downward trend in time-to-resolve (190, 68, then 24 minutes) as a demonstration of the mechanism and not as a field result. I have not measured this on real production incidents.
What I can measure is behavior. The evaluation script runs 29 scenarios across 10 fields and checks root cause, category, symptoms and, importantly, that no unsafe step appears. CI fails the build if accuracy drops or an unsafe step shows up. Each resolve also grades the engine's own prediction (correct, partial or incorrect) against the confirmed cause, and the confidence number only turns into a calibrated probability after enough graded outcomes. Until then the API says plainly that it is a ranking score, not a probability.
Lessons
- Store the failures. The dead ends are the most reusable part of an incident. Give them equal weight in memory.
- Never trust "it worked". Classify a fix before you save it. A restart that recovers the service is relief, not a repair.
- Make memory render as sentences. If your memory layer is text based, give your records a natural-language form on purpose.
- Put a seam around memory. A two-method protocol let me test everything without a network, and it keeps the report honest about which backend answered.
- Treat recalled text as untrusted input. Anything a human typed once can end up in a prompt later.
The part I would change first is the regex classifier. It is transparent, but it will miss phrasings I did not think of. The next step is to keep the rules as the safe default and let recalled history from Hindsight correct them over time. If you are weighing how to give an agent durable memory, what agent memory is is a good place to start.
Top comments (0)