I Made Incident Memory Show Its Work

# I Made Incident Memory Show Its Work

An incident assistant that remembers past outages can be confidently wrong in a more convincing way than one that forgets. I built this system around a stricter idea: if memory changes the recommendation, the engineer should be able to inspect which past incident changed it and why.

The system takes an alert and produces two answers. The first sees only the alert. The second can use similar incidents and engineer feedback stored in Hindsight. The point of showing both is to make the effect of memory visible: a generic suggestion on one side, and a recommendation grounded in evidence on the other.

The path from alert to evidence

The [Streamlit incident comparison UI]([][(https://github.com/bharat24082007-spec/incident-response-agent/blob/main/app.py)] collects an alert and asks agent.py for both answers. For the memory-informed answer, the agent calls memory.py, which uses the Hindsight Python client to recall relevant incident records. The agent then sends the alert and those records to Groq.

The memory path is deliberately explicit:

Streamlit UI -> agent.py -> Groq
                    |
                    v
                memory.py -> Hindsight

seed_data.py provides the initial incident history, and load_memory.py stores it in Hindsight’s incidents bank. After an engineer evaluates a recommendation, app.py passes the outcome to memory.retain_outcome(), which saves it to the same bank. A later alert can recall that feedback. Hindsight’s implementation is available in the Hindsight GitHub repository.

That last step is what makes this more than retrieval over a static incident archive. The system has a place to record what an engineer says happened after trying a suggestion, and a path for that result to influence subsequent answers. This is the practical role of [Vectorize agent memory][(https://vectorize.io/what-is-agent-memory)]

Retrieval is a candidate list, not proof

The first design decision was to treat Hindsight recall as a way to find candidates, not as a verdict. A search result can look similar because it shares a service name or a common symptom. That does not make it evidence for the current root cause.

The memory wrapper asks Hindsight for source chunks along with recall results. That matters because a short extracted fact may not include the config value or failed workaround that made the incident useful. The code keeps the original record text available to the agent:

response = _hindsight_call(
    "recall",
    bank_id=BANK_ID,
    query=alert_text,
    max_tokens=2048,
    budget="high",
    include_chunks=True,
    max_chunk_tokens=8192,
)

This follows the Hindsight Python client’s recall API, including its support for source chunks. memory.py parses the returned record into fields such as service, alert, cause, working fix, failed fixes, and resolution time. It also preserves relevance rank and score for display. Those scores are relative signals from the search, not calibrated probabilities, so the prompt tells the model not to treat a high score as certainty.

The agent limits the candidate records sent to Groq to five. It asks the model to include only incidents it considers closely or partially relevant, with a match reason. Then _prepare_answer() checks that every cited ID was actually among the supplied records and discards evidence labeled weak. The model cannot invent a sixth incident and have it appear in the evidence list.

for item in answer.get("Evidence from past incidents") or []:
    incident_id = str(item.get("id", ""))
    strength = str(item.get("match_strength", "weak")).lower()
    if incident_id not in by_id or strength not in {"close", "partial"}:
        continue

This is a small boundary with an important consequence: the model can explain the evidence, but the application owns which records are valid sources. The UI then caps the evidence section at three items and displays service, date, fixes, failed approaches, and time to resolve. That gives an on-call engineer something concrete to challenge instead of asking them to trust a fluent summary.

Keeping the no-memory answer honest

A side-by-side comparison only means something if the two sides have different information. The no-memory prompt therefore receives the alert text alone and is instructed not to assume anything about the company’s services or infrastructure. Its confidence is capped at medium. It cannot accidentally benefit from the memory path through shared company context.

For the memory answer, the system prompt says to ignore weak or unrelated records, cite the exact ID and date, and base confidence on actual match strength. High confidence requires at least one close match. Without relevant evidence, the answer’s confidence is low.

This is a useful division of responsibility. Groq proposes a diagnosis and selects evidence from a bounded set. The Python code verifies that the citations point to real recalled records. The engineer sees both the proposal and the supporting material.

The primary model is openai/gpt-oss-120b, with openai/gpt-oss-20b as fallback. Each model gets up to three attempts, and the client has its own timeout configured. The Groq chat completion quickstart covers the request interface used by the agent. I keep retry handling outside the memory wrapper because a Groq request failure and a Hindsight retrieval failure are different failures, with different effects on the answer.

Feedback needs its own semantics

The other design decision was not to treat feedback as another seeded incident. Feedback is newer, has a different source, and should be visible as such. Outcome records include metadata such as record_type, worked, service, title, alert, date, and note. Metadata values are converted to strings before retention because Hindsight expects string values.

The agent gives learned feedback higher priority when it matches the alert, while also instructing the model not to let recent but unrelated feedback override a clearly relevant older incident. Recency matters, but relevance still gates whether the memory applies.

I also made the outcome record conservative. If an engineer leaves the notes box empty, the system stores an explicit confirmation rather than copying the model’s suggested steps and presenting them as an action the engineer performed. The engineer supplies a required resolution time separately. Repeated submissions with the same alert and note are skipped, and the feedback buttons disable after a successful save until the next comparison.

The core retention call is small; the choices around its content are what make the record useful:

return _hindsight_call(
    "retain",
    bank_id=BANK_ID,
    content=content,
    context="Northstar Commerce production incident outcome",
    document_id=outcome_id,
    metadata=_string_metadata(
        incident_id=outcome_id,
        service=service,
        title=title,
        date=date.today().isoformat(),
        record_type="outcome",
        worked="true" if worked else "false",
        note=submitted_note,
        alert=alert.strip(),
        time_to_resolve_minutes=time_to_resolve,
    ),
)

In the complete service, I would also treat the outcome write as an auditable operational event: who submitted it, when, and which suggestion version it answered. The current record already separates the alert, outcome, note, and resolution time so those fields can be extended without asking the model to reconstruct them later.

The event loop was part of the integration

One implementation detail caused a less interesting but very real failure: the synchronous Hindsight client uses asyncio internally, while Streamlit runs inside an event loop. Calling the synchronous method directly produced “This event loop is already running.”

I moved each Hindsight call into a fresh worker thread and constructed the client in that worker. The wrapper stays small:

def run_in_thread(fn, *args, **kwargs):
    with ThreadPoolExecutor(max_workers=1) as executor:
        return executor.submit(fn, *args, **kwargs).result()

The Hindsight Python SDK documentation describes both sync and async methods. In this integration, the thread boundary lets the existing synchronous interface run outside Streamlit’s loop. It also gives the UI a single place to route retain, recall, bank setup, and outcome writes. The tradeoff is straightforward: the Streamlit request waits for the worker’s result. For this interaction, correctness and a clear control flow were more useful than adding another async layer.

A recalled feedback record appears above the engineer feedback controls.

Consider an alert about payments-gateway rejecting authorizations while Redis approaches its client limit. Without memory, the agent can explain connection exhaustion in general terms. With memory, it can cite a prior payments-gateway incident where a large per-replica pool caused aggregate connection demand to approach the Redis limit. The evidence can name the earlier pool setting, the change that resolved it, and the approaches that failed.

Now consider a new schema-related consumer failure with no matching incident. The memory path should not force an unrelated Kafka incident into the explanation just because both involve a consumer. It can return no evidence and low confidence. If an engineer later records the actual cause and resolution, a similar alert can recall that outcome as learned feedback. That gives the engineer a way to see what changed between the first and later response.

This behavior is the reason the UI keeps the two answers side by side. A memory system that only makes the answer sound more specific is difficult to evaluate. Showing the alert-only response, the memory-informed response, and the citations makes the difference inspectable.

What I took from building it

• Retrieval quality includes refusal. A useful memory system needs to leave weak matches out, even when it has results to show.
• Citations need an application-side boundary. Asking a model to cite records is not enough; the code should validate cited IDs against the supplied records.
• Engineer feedback is not model output. Store what the engineer confirmed, and keep it distinguishable from older incident history.
• Operational integrations need thread and retry boundaries. Streamlit’s event loop and provider failures are part of the system design, not incidental plumbing.
• A comparison view is an evaluation aid. The no-memory answer makes it easier to spot when memory adds concrete evidence and when it adds nothing.

The main lesson for me is that long-term memory earns its place when an engineer can trace a recommendation back to a record and correct it with a durable outcome. Recall supplies candidates. The application checks the evidence boundary. The engineer remains responsible for deciding whether the proposed fix is safe to try.