My Incident Agent Forgot Every Outage Until I Gave It Hindsight

Likhith.R — Sun, 08 Nov 2026 18:25:25 +0000

Every time I restarted my incident agent, it forgot everything it had learned. I'd teach it about an outage, close the notebook, and come back to a blank slate with four sample incidents. This article covers how I fixed that with Hindsight, and what still doesn't work.

The agent and its memory problem

The agent takes a live incident (a title, a service, the symptoms, and a severity), recalls similar past incidents, and returns a five-part response plan: likely cause, immediate steps, the memories it relied on, how to verify the fix, and when to escalate. I call it INFERON AI. The model is openai/gpt-oss-120b on Groq, and a Gradio form makes it usable from a browser.

The INFERON AI web form with a checkout incident entered as SEV1. Clicking "Get response plan" returns a five-section plan, which appears under the form.

My first version stored incidents in SQLite and matched them with TF-IDF. It worked within one session. Adding a postmortem or a resolved incident made the plans better, and I could see the effect right away.

Then the session ended. Colab deletes its files when a session closes, so the database went with it. The next session started with only the sample incidents I had hard-coded. Every outage I had taught the agent was gone, and so was every postmortem lesson. The agent was only as good as the code I'd typed, not what it had learned.

That's the real failure. Keyword matching was a weaker problem, and I'll come back to it. What hurt was that the agent couldn't keep anything.

Moving the memory to Hindsight

I moved storage and recall to Hindsight, a memory service with a cloud API. Each incident becomes one plain-text record in a memory bank called incident-memory. Hindsight extracts the facts from that text on its own, so I don't split the fields myself:

def incident_text(inc):
    return (f"Incident: {inc['title']}. Service: {inc['service']}. Severity: {inc['severity']}. "
            f"Symptoms: {inc['symptoms']}. Root cause: {inc['root_cause']}. "
            f"Resolution steps: {inc['resolution_steps']}. Runbook: {inc['runbook']}.")

Recall asks the bank for the facts closest to the live incident:

def recall_similar(title, service, symptoms, top_k=8):
    query = f"{title}. Service: {service}. Symptoms: {symptoms}"
    resp = in_thread(hs.recall, bank_id="incident-memory", query=query)
    return resp.results[:top_k]

I first set top_k to 3, but one query returned 57 facts, since Hindsight breaks each incident into several statements. Three results could all come from one incident, so I raised the limit to 8.

A real recall from the incident-memory bank. Hindsight returns several facts for each stored incident, so the results include near-duplicates.

The difference shows up across sessions. After I restarted the notebook, I didn't reload the seed data, and the agent still recalled the stored incidents. I also recorded a resolved payments incident and a postmortem, and those stayed in memory too. For the first time, the agent kept what I taught it.

INFERON AI sends the incident to Hindsight for recall, then sends the incident and memories to the Groq model. Postmortems and resolved incidents go back to memory with retain.

What I had to fix along the way

The event loop broke the web form. The Hindsight client conflicted with Colab's event loop. The common fix, nest_asyncio, made the client work but broke Gradio's server, which then failed to start. Restoring the original function didn't help, because the server had already captured the patched version. The fix that worked was to stop patching the event loop and run every Hindsight call in a single worker thread, which has no running loop:

_pool = concurrent.futures.ThreadPoolExecutor(max_workers=1)
def in_thread(fn, *args, **kwargs):
    return _pool.submit(fn, *args, **kwargs).result()

The account needed credit. Before any of this worked, the connection returned 402 Payment Required. The API was reachable, but the account had no balance. Check that before you build anything on a paid memory service.

What Hindsight doesn't do yet

Persistence solved the problem I came in with, but it didn't solve everything.

Duplicates crowd the results. I seeded the same incidents more than once while testing, and the recall output shows the result. Several records for the same checkout incident appear in a single query, with slightly different wording. When the top results are copies, fewer distinct incidents fit in the model's context. If you load your seed data more than once, check the bank for duplicates before you trust a plan.

The model chose among near-duplicates. On the SEV1 run, the plan quoted a stored record word for word. On the SEV3 run, it quoted a different record for the same incident, which included the phrase "due to lack of entry eviction." That phrase was in memory, not invented by the model. The model picked one of several versions, and nothing in the output showed that a choice had been made.

The SEV1 plan. Section 3 quotes a stored memory word for word, but the memory bank holds several near-identical versions of the same incident.

Outcomes don't reorder results. My SQLite version multiplied each match's similarity by how often its fix had worked. A fix that held could outrank a closer match that failed. In the Hindsight version I stored outcomes as text, and they don't change the ranking. When I recorded a payments rollback that had worked, a checkout query still returned checkout memories first. Raising the limit let the payments outcome into the top eight, but it never ranked first.

The model adds details that aren't in memory. The SEV1 plan suggested a POST /admin/clear-retry-cache endpoint and a payment_retry_cache_entries metric. Neither appears in any stored record. Some of that is reasonable general advice, and some is plausible-sounding invention, so every step needs checking against your real systems.

The model mixed up versions. The stored fix was a rollback to v2.3. The plan recommended v2.6, which was the version in the live incident. Nothing in the output flagged the difference.

Severity changes wording more than actions. SEV1, SEV2, and SEV3 plans had nearly the same steps, with different escalation text.

What I'd do next

The memory now survives between sessions, which was the reason I started. The next problems are ranking outcomes, removing duplicates, and checking the model's output.

I'd store success rates outside the memory system and combine them with recall results, the way the SQLite version did. I'd deduplicate records when they're stored, so the top results cover different incidents. I'd also run a check after the model responds that compares each quoted line with the retrieved memories, and rejects any quote that doesn't match. And I'd make SEV1 change the actions, not just the wording, so it pages the on-call lead before any other step.

Lessons

Test persistence before you test intelligence. My agent looked smart inside one session, and it was useless across sessions. I only caught that by restarting the notebook.
Memory that lives in a notebook's files is not memory. Anything stored in a session's local disk disappears when the session ends.
More results is not the same as better ranking. Raising the limit brought the right memory into the context, but didn't move it to the top.
Duplicates change what the model sees. Repeated records push out other incidents, and the model may quote whichever copy it reads first.
Keep blocking calls out of your web server's event loop. Put them in a worker thread instead.

Hindsight gave the agent a memory that outlasts the session. Deciding which memories should win, removing duplicates, and checking what the model says about them is still my job. You can read more in the Hindsight documentation and in Vectorize's guide to agent memory.