How Hindsight Lets an Incident Agent Remember Past Outages

How Hindsight Lets an Incident Agent Remember Past Outages

Every on-call engineer has had the same déjà vu: a new alert fires, the symptoms look familiar, and somebody says "didn't we see this in the spring?" Then comes twenty minutes of searching chat history for the answer. We wanted an AI agent that could do that remembering for us, and we quickly learned that "give the model more context" does not get you there on its own.

IncidentIQ is the prototype we built to explore this. It investigates a production incident with a Groq model, asks an engineer to verify the real cause, and stores only the verified outcome in Hindsight, an open-source agent memory system. This article is about the memory half of that design: what we store, how we retrieve it, and what the agent does with it. The incidents are synthetic and the system runs locally.

Why a bigger prompt is not memory

The obvious baseline is to paste last month's postmortems into the prompt. That breaks quickly. The context window fills up, nothing decides which postmortem is relevant, and the model has no way to tell a confirmed cause from a guess made halfway through an incident.

The second obvious baseline is a SQL query. PostgreSQL stores every incident we create, and it remains our record of truth, but it is a poor tool for "have we seen something like this?" An old incident might say the connection pool was exhausted. A new one might show a QueuePool limit reached error from a different service. A LIKE query matches neither to the other.

Hindsight sits in between. As described in the Hindsight documentation, retaining content extracts structured facts and the entities in them, and recall runs several retrieval strategies in parallel, namely semantic similarity, keyword, graph traversal and temporal, then merges and reranks the results. Vectorize's overview of agent memory explains the broader idea. For us the practical consequence is that we write one natural-language record per verified incident and let the memory system decide what is retrievable later.

One clarification, because people often assume otherwise: nothing here trains or fine-tunes the model. Groq's model is exactly the same on every call. What changes is the context we hand it.

What we retain

We keep three kinds of memory apart. A verified resolution is created only after an engineer confirms a root cause and a fix. A failed approach records something that was tried and did not work. The service also has a method for retaining raw incident observations, but the application never calls it. Symptoms alone say nothing about the cause, so we did not want them in memory.

A verified resolution is assembled as plain text, with credential-shaped strings redacted first:

content_parts = [
    "Engineer-verified resolution",
    f"Verified by: {_clean_text(verified_by)}",
    f"Verified root cause: {_clean_text(root_cause)}",
    f"Resolution: {_clean_text(resolution)}",
]
if incident_id:
    content_parts.append(f"Incident: {_clean_text(incident_id)}")
if technical_context and technical_context.strip():
    content_parts.append(f"Technical context: {_clean_text(technical_context)}")

Including the incident number and the service in the text matters more than it looks. Because Hindsight extracts facts from the text, those details are what let a later recall say which incident a memory came from.

The write itself is one call:

response = await self._client.aretain(
    bank_id=self._bank_id,
    content=content,
    context=context,
    tags=list(tags),
    document_id=document_id,
    timestamp=_format_timestamp(occurred_at),
    update_mode="replace",
)

Three arguments do real work. tags (incident and verified-resolution) let us tell resolutions from failed approaches later. timestamp records when the event happened rather than when we happened to retain it. And document_id is where our idempotency lives. We derive it from a hash of the memory type, incident and content:

def _stable_document_id(memory_type, incident_id, content):
    identity = "\0".join((memory_type, incident_id or "", content))
    digest = hashlib.sha256(identity.encode("utf-8")).hexdigest()[:32]
    return f"incidentiq-{memory_type}-{digest}"

Submitting the same verification twice therefore produces the same document, and update_mode="replace" overwrites it instead of stacking duplicates. The UI reports that case as skipped_duplicate rather than pretending it was a fresh write.

How recall feeds the agent

Before a MEMORY_ON investigation reaches the model, the service builds a query from the incident description and its technical context, then recalls:

response = await self._client.arecall(
    bank_id=self._bank_id,
    query=query,
    budget="low",
)
memories = tuple(
    HistoricalMemory(
        id=result.id,
        text=result.text,
        memory_type=result.type,
        context=result.context,
        occurred_start=result.occurred_start,
        mentioned_at=result.mentioned_at,
        tags=tuple(result.tags or ()),
    )
    for result in response.results
)

Each HistoricalMemory carries a fixed provenance string, historical_hindsight_memory. That label travels all the way to the prompt and to the interface, so a recalled fact never gets mistaken for something observed in the current incident. In the prompt they get their own references (historical:0, historical:1), and the model is told that historical similarity is not proof of a current cause and that it may cite only the references it was given.

Before and after

Take two synthetic payment-api incidents. In the first, checkout requests return HTTP 503 and database connection use is near its ceiling. With memory off, the model can only reason from the logs and metrics in front of it. An engineer then verifies that a worker-concurrency increase exhausted the connection pool, and that fact is retained.

In the second incident, the checkout failures return after a later release. With MEMORY_ON, recall can hand the model the earlier verified cause as a labelled historical memory, so concurrency and pool sizing become hypotheses worth checking early. [INSERT: the real recalled memory text or a cited memory from your own run.] The model still has to reconcile that with the new deployment notes, and it can still be wrong. The difference is in what it starts from.

We have not benchmarked retrieval quality, and we do not claim a measured gain in accuracy or resolution time.

Lessons

Store conclusions, not transcripts. A short, verified statement of cause and fix is a better memory than a long chat log.

Put identifiers in the text. Incident numbers and service names make recalled facts traceable.

Make writes idempotent. A deterministic document_id plus replace semantics removed a whole class of duplicate-memory bugs.

Keep provenance attached to every memory. The model, the UI and the postmortem all need to know where a fact came from.

Treat memory as context, never as proof. That rule belongs in the prompt and in the interface.

Limits

Recall currently uses a low budget and a single query, and we do not attach structured metadata such as the incident number to the write. A successful retain tells us the API accepted the content. We have not yet measured how quickly new facts become recallable, which is something to quantify before relying on the loop in a real on-call rotation. Memory poisoning is also a real risk, which is why retention is gated on human verification.

The repository is on GitHub if you want to read the services yourself.