My On-Call Agent Remembered the Rollback That Never Works

NIKITHA THAMMAIAH — Sat, 07 Nov 2026 07:49:32 +0000

The alert said checkout-api p99 4.8s, HikariCP 50/50, v3.18.2 deployed 12 minutes ago. I gave it to the same LLM twice. The first answer, step four: "roll back the deployment." The second answer put rollback at the top of its Avoid list: "Rollback attempts in INC-2291 and INC-2347 did not improve the incident and were aborted."

Same model, same prompt shape, same alert. The only difference was that the second call had memory.

That gap is the whole reason I built Déjà Vu, an on-call agent that keeps our incident history in Hindsight agent memory and gets better every time someone closes an incident. This post is about what it took to make that answer show up reliably, because the first version didn't.

The problem is not missing knowledge

Every on-call team I've worked with has the answer to tonight's outage written down somewhere. It's in a postmortem from two months ago, in a Slack thread, or in the head of the engineer who's on leave. The failure mode isn't that nobody knows. It's that at 2 AM nobody remembers, and the reflexes take over: roll back the last deploy, scale up, restart pods.

Those reflexes are usually fine. For one specific class of incident they're poison. Our checkout service had the same pool-exhaustion incident twice:

INC-2291: a payment SDK bump raised the HTTP read timeout from 3s to 30s. When the bank API slowed down, request threads held database connections while they waited, and the Hikari pool ran dry. The on-call rolled back checkout (14 minutes, no change), then scaled from 24 to 40 pods, which made connection pressure worse. Time to mitigate: 52 minutes.
INC-2347: same symptoms, new trigger (three retries on a 3s timeout). The responder started a rollback again before checking gateway latency, and aborted after 10 minutes once a senior engineer remembered INC-2291.

A stateless LLM can't help here. It will give textbook advice, and the textbook says roll back the recent deploy. What I needed was an agent that knows our incidents, including the parts nobody writes in the summary: what we tried that didn't work.

How it hangs together

The system is small on purpose:

Browser UI ──► FastAPI
                ├── agent.py   prompt building, output checks
                ├── llm.py     Groq (gpt-oss-120b), JSON mode
                └── memory.py  Hindsight: retain · recall · reflect

Hindsight does the memory work. There are three operations, and each maps to a moment in the incident lifecycle:

Moment	Hindsight call	What happens
Postmortem published	`retain`	Incident stored with its real timestamp
Alert fires	`recall`	Similar incidents and failed attempts retrieved
Incident closed	`retain`	What fixed it, what wasted time, whether the agent helped
"What keeps going wrong?"	`reflect`	Reasoned answer across the whole history

I picked Hindsight over a homegrown vector store for one reason: I didn't want to build fact extraction, entity linking and temporal search myself. When you retain a postmortem, Hindsight splits it into typed facts (world facts, experiences) and then consolidates them in the background into observations, which are beliefs backed by evidence. Its recall runs semantic, keyword, graph and temporal search in parallel and reranks the results. The Hindsight docs explain the internals. What mattered to me is that a query about "checkout latency" can reach a fact about "Hikari pool exhaustion" even though the two share no keywords.

The bank gets a mission when it's created, which shapes how reflect reasons:

MISSION = (
    "I am the on-call memory for an e-commerce platform's SRE team. I track incidents, their symptoms, "
    "root causes, the fixes that worked and the actions that wasted time. I prefer fixes verified in past "
    "incidents over generic advice, and I notice when the same failure pattern comes back with a new trigger."
)

self.client.create_bank(
    bank_id=self.bank,
    name="On-call incident memory",
    mission=MISSION,
    disposition={"skepticism": 4, "literalism": 3, "empathy": 2},
)

Skepticism is at 4 deliberately. An incident agent that believes everything it reads is worse than none.

The first version found the incident and missed the lesson

My first triage did one broad recall: "Incidents on checkout-api or with similar symptoms… what was the root cause, what fixed it, and what did not help?"

It matched INC-2291 and INC-2347 right away, with high confidence and the correct root cause. It also returned 56 memories in 5.3 seconds, most of them about cart-svc DNS caching, pricing caches and TLS certificates. The facts I cared about most, "rolled back checkout-api, no improvement", weren't in the results at all. So the agent recommended the right fix and still didn't warn against the rollback. Worse, it invented CLI commands (ffctl set pg_circuit_breaker=true) that exist nowhere in our tooling.

I made three changes, and each one taught me something about working with memory.

1. Failed attempts are their own memories

In the postmortem text, the failed rollback was one clause inside a long paragraph. Extraction is good, but that clause was competing with symptoms, root cause and fix for attention. So the seed step now retains failed attempts as separate documents:

if inc["failed_attempts"].strip() and not inc["failed_attempts"].lower().startswith("none"):
    mem.retain(
        f"During incident {inc['id']} on {inc['service']}, these actions did NOT help or made it worse: "
        f"{inc['failed_attempts']} Next time on {inc['service']}, do not try this first.",
        context="failed mitigation attempt",
        document_id=f"{inc['id']}-failed",
        timestamp=datetime.fromisoformat(inc["date"]),
        metadata={"service": inc["service"], "kind": "failed_attempt"},
    )

After a re-seed (Hindsight only charges for changed content, so re-sending the unchanged postmortems cost nothing), consolidation produced observations like "Do not attempt a rollback of checkout-api v3.16.1 as a first step during future incidents." That isn't a sentence I wrote. Hindsight derived it from the evidence.

2. Two narrow recalls beat one broad one

One query was trying to answer two different questions. I split it:

def build_queries(alert: Alert) -> list[str]:
    return [
        f"{alert.service} incident with symptoms like: {alert.text} Root cause and fix that worked.",
        f"Actions that did NOT help or made it worse during incidents on {alert.service}.",
    ]

The two recalls run in parallel with the baseline LLM call, at different budgets (mid for symptoms, low for failures). Recall dropped from 12 seconds to about 6, and the result set shrank from 56 memories to 23 relevant ones. Both rollback failures now show up in every checkout triage.

3. Memory about other services is a liability

Then I triaged an alert the agent had never seen: recommendation-svc OOMKilled 14 times after a new embedding model rollout. With memory on, it produced three confident Avoid items, citing checkout-api, inventory-svc and platform-k8s incidents. The recall had faithfully returned the nearest memories, and the LLM had faithfully used them. Nothing in that chain was technically wrong, and the output was still nonsense.

The fix is one function:

def same_service(memories: list[dict], service: str) -> tuple[list[dict], int]:
    kept = [m for m in memories if service.lower() in m["text"].lower()]
    return kept, len(memories) - len(kept)

The UI now prints "30 about other services ignored" in the memory trace header, so it's clear the agent saw them and chose not to use them. I also reject any incident ID in the LLM's output that never appeared in the recalled memories.

What it looks like now

Checkout alert, before closing the loop. Without memory: "roll back the deployment," "consider scaling pods." With Hindsight: matches INC-2291 and INC-2347; Do first: check gateway latency, enable pg_circuit_breaker, set pg_client_retries=0; Avoid: rollback (failed twice), scaling to 40 pods (made it worse).

After I closed the incident. I recorded the resolution in the UI: circuit breaker plus zero retries, rollback skipped, 9 minutes. The API retains it:

memory.retain(
    agent.resolution_to_text(res),
    context="incident resolution reported by on-call engineer",
    document_id=res.incident_id,
    timestamp=datetime.now(timezone.utc),
    metadata={"service": res.service, "kind": "resolution", "helpful": str(res.suggestion_helpful).lower()},
)

On the next triage of the same alert, my resolution came back as the first recalled memory and the top match. Confidence went from medium to high. Do first reordered so the proven fix, the circuit breaker, became step one instead of "check the read timeout." Nobody edited a runbook.

The never-seen alert. For recommendation-svc, the agent with memory returned low confidence, seen before: none, avoid: none. The agent without memory returned high confidence and cited "past incidents where manual pod deletion delayed resolution", incidents it cannot possibly know about. That contrast surprised me more than anything else. A stateless model doesn't just lack history; it invents it.

Asking the memory directly. reflect("Which first actions have repeatedly wasted time during checkout incidents?") came back with both rollbacks, how long each one cost (14 and 10 minutes), the scaling mistake, the shared root cause and the fixes that worked. Every claim traced back to a retained fact. This is what a new on-call engineer should read before their first sale-night shift.

Lessons I'd reuse

Retain the negative space. The most valuable memory in incident response is what didn't work, and postmortem prose buries it. Give failed attempts their own retain call and their own document ID.
One recall per question. A broad query lets loosely related memories crowd out the one fact you need. Two narrow queries in parallel were faster and more precise than one big one.
Scope memory before you trust it. Recall returns the nearest memories, not the relevant ones. Filter by the entity you're actually reasoning about, and show what you dropped.
Make the baseline visible. Running the same alert with and without memory, side by side, is the best debugging tool I had. It's also how I noticed the stateless model inventing incidents.
Close the loop in the product, not in a wiki. The learning comes from one retain call at resolution time, with a "did the agent help?" bit. If that step takes more than 30 seconds, nobody will do it.

One honest pain point: the synchronous Hindsight Python client binds its HTTP session to the event loop of the first thread that uses it. When I parallelized the recalls, the next retain call failed with a 500. The fix was to give each client its own dedicated thread and route every call to it through that thread. It's a small detail, but it cost me an hour.

The same-service filter is also a trade-off I haven't fully resolved. Some lessons do transfer: "don't scale consumers during a rebalance storm" applies to any Kafka service. A smarter version would filter by failure pattern instead of service name, which is something Hindsight's observations and agent memory model make possible. For now, precision wins.

The bigger takeaway is that a memory layer doesn't replace on-call judgment. It replaces the senior engineer who happened to remember INC-2291, and makes that memory available to whoever is holding the pager tonight.

DEV Community: NIKITHA THAMMAIAH