I Stopped Treating Every Incident as a New Problem

Kiran V — Sat, 07 Nov 2026 10:45:15 +0000

I Stopped Treating Every Incident as a New Problem

Building an incident response agent with persistent memory using
Hindsight, FastAPI, and MongoDB.

The frustrating thing about incident response is that the same failure
can happen twice, sometimes months apart, and the investigation can
still start from zero.

Someone notices elevated latency. Someone else checks the database. The
team searches old tickets, messages, dashboards, and notes. Eventually,
somebody remembers: Didn't we have something similar last month?

That knowledge exists, but it is scattered across incident records and
the people who remember them.

I wanted to explore a different approach: what if an incident response
agent could retain useful experience from previous incidents and recall
it when a new problem appears?

That question led me to build an Incident Response Agent using FastAPI,
React, MongoDB, and
Hindsight.

The interesting part was not simply connecting an API to a language
model. It was thinking about the difference between storing incident
history and giving an agent a way to use that history later.

The problem: incident history is not the same as memory

Most incident management systems are good at recording what happened.
They can store titles, descriptions, severity, timestamps, resolution
notes, and status changes.

But storing an incident does not automatically make its lessons useful
during the next investigation.

Imagine two incidents:

Incident A: The payment API becomes unavailable after database connection usage increases.
Incident B: Several weeks later, the payment API starts returning intermittent errors, and database connections are again approaching their limit.

A conventional workflow might treat Incident B as an entirely new
investigation. An engineer has to search for Incident A, read its
resolution, and decide whether the old information applies.

I wanted the agent to have a more direct path: retrieve relevant
historical experience while investigating the new incident.

That is where Hindsight became central to the design.

The architecture: a familiar application with a memory layer

I kept the application architecture deliberately straightforward.

React provides the dashboard for viewing and interacting with incidents.
FastAPI exposes the backend API and coordinates incident operations.
MongoDB stores structured incident records and application data.
Hindsight provides the memory layer for retaining and recalling incident experience.
The incident analyzer produces suggestions from incident information.

The distinction between MongoDB and Hindsight matters. I use MongoDB for
application records that need predictable structure and retrieval.
Hindsight serves a different purpose: making retained experience
available for later recall.

The flow looks like this:

An incident enters the application.
The backend stores its structured record.
The agent can retrieve relevant historical context through Hindsight.
The analyzer evaluates the incident and produces suggestions.
When an incident is resolved, its useful experience can be retained for future investigations.

This separation keeps the application data model understandable while
giving the agent a dedicated memory capability.

Incident dashboard

The overview screen summarizes incident counts and shows the active-incident panel, which is populated from MongoDB.

Why I chose Hindsight

I did not want to treat memory as another database table that the agent
could query with a few filters.

A useful agent memory system should help connect a current situation to
previously retained information, even when the wording is different.

For example, an earlier incident might mention database connection pool
exhaustion, while a new incident describes requests waiting for
available database connections. Those descriptions are not identical,
but they may point toward related operational experience.

Hindsight provides a retain-and-recall approach that fits this problem.

I used the Hindsight agent memory
project and its
documentation as the foundation for
this part of the system. The broader idea is that an agent should be
able to accumulate and retrieve experience rather than depending
entirely on the context of one request. The Vectorize explanation of
agent memory is useful
background for understanding this distinction.

For me, the important design decision was to keep memory operations
separate from the incident's ordinary CRUD operations. Creating an
incident and remembering a resolved incident are related, but they are
not the same operation.

Retain and recall: the important lifecycle

I think about the memory lifecycle in two stages.

Retain: Once an incident has been investigated and resolved, the
system can retain useful information about the event. This might include
the symptoms, affected service, suspected cause, resolution, and
operational lessons.

Recall: When another incident arrives, the system can search for
relevant retained experience and make that context available to the
investigation workflow.

The following is a conceptual illustration of the workflow, not a
verbatim repository excerpt:

def investigate_incident(incident):
    historical_context = recall_relevant_incidents(
        incident["title"] + " " + incident["description"]
    )

    analysis = analyze_incident(
        incident=incident,
        historical_context=historical_context
    )

    return {
        "analysis": analysis,
        "historical_context": historical_context
    }

The key idea is that the new incident is not evaluated in isolation.
Historical context can be retrieved and presented alongside the current
incident.

After resolution, a separate operation can retain the experience:

def remember_resolved_incident(incident):
    experience = {
        "title": incident["title"],
        "description": incident["description"],
        "service": incident["service"],
        "severity": incident["severity"],
        "resolution": incident.get("resolution")
    }

    retain_incident_experience(experience)

I prefer this separation because it makes the lifecycle explicit. A
record being saved to MongoDB does not automatically mean it has been
retained as agent memory. Likewise, retrieving a memory does not mean
the system should blindly treat it as the cause of the current incident.

Before publishing: Replace these conceptual examples with the actual
retain and recall calls from the repository if you want to present them
as implementation excerpts. Keep the label if you retain the simplified
versions.

Incident records

The incidents screen lists records available in the application, including severity, status, and associated service.

AI incident diagnosis

The AI Diagnosis screen lets an operator select an incident and start an analysis using rule-based logic and Hindsight historical memory.

The lesson: retrieved memory is evidence, not truth

One of the most important engineering decisions in this project was
keeping historical context separate from the actual analysis result.

A previous incident can be relevant without being identical. If a
database problem caused an outage last month, that does not prove the
database is responsible for today's outage.

The agent should use recalled information as investigation context, not
as unquestionable truth.

The current analyzer uses rule-based logic to generate suggestions from
incident information. Hindsight retrieval provides historical context,
but retrieval alone does not establish a verified root cause.

I want that distinction to remain visible in the application:

A historical incident is a reference, not proof.
A rule-based suggestion is a hypothesis, not a diagnosis.
A resolved incident is useful experience, but its lesson may not apply to every future event.

This is also why I avoid presenting a confidence score as if it were a
calibrated probability unless it has actually been evaluated that way.

What the application exposes

I built the backend around explicit incident operations rather than
hiding everything behind one large endpoint.

The API includes operations for creating and listing incidents,
searching memory, analyzing an incident, resolving it, and retaining its
experience.

The intended route structure is:

POST  /incidents/
GET   /incidents/
POST  /incidents/memory/search
POST  /incidents/{incident_id}/analyze
PATCH /incidents/{incident_id}/resolve
POST  /incidents/{incident_id}/memory/store

Before publishing: Verify each path and HTTP method against the
current FastAPI route definitions. Remove or correct any route that does
not exist in the final repository.

This makes it easier to test the application and understand where each
operation belongs.

The React dashboard provides a visual entry point for incident
information. FastAPI handles the application workflow, while MongoDB and
Hindsight serve their separate persistence and memory responsibilities.

The project also pushed me to think about operational details that are
easy to overlook in an early implementation: error handling, database
availability, API authentication, input limits, and the difference
between a working local integration and a production-ready service.

An agent that investigates incidents is itself part of an operational
system. It needs to fail safely and communicate what it actually knows.

System architecture

The architecture separates structured incident storage from persistent agent memory. FastAPI coordinates both services and passes the available incident context to the rule-based analyzer.

What I learned building with memory

A few lessons stood out.

1. Persistent memory needs a lifecycle

It is not enough to add a memory API call and declare that an agent has
learned something. I had to think about when information should be
retained, what information is useful, and when it should be recalled.

2. Application storage and agent memory solve different problems

MongoDB gives the application structured records. Hindsight gives the
agent a way to retain and recall experience. Treating them as
interchangeable would make the architecture harder to reason about.

3. Recall should influence investigation, not dictate the answer

A similar incident can save investigation time, but it should not
automatically determine the root cause. Historical information needs to
be evaluated against the current symptoms.

4. Explicit operations make an agent easier to debug

Separating incident creation, analysis, memory search, resolution, and
memory retention makes the system easier to inspect and test. When
something goes wrong, I can narrow down which part of the workflow
failed.

5. A credible agent must be honest about its capabilities

There is a difference between retrieving a relevant incident, generating
a rule-based suggestion, and proving a root cause. Keeping those
concepts separate makes the system more trustworthy.

Where I want to take it next

The natural next step is to make the investigation workflow more deeply
memory-aware.

I want to evaluate whether recalled incidents actually help narrow
investigations, improve the relevance of suggestions, and reduce
repeated work. That requires a repeatable test set of incident
scenarios, clear evaluation criteria, and careful comparison between
investigations with and without historical context.

I also want to strengthen the operational foundation: authenticated API
access, reliable persistence, better observability, and a dashboard
connected to real incident workflows.

The goal is not to make an agent that confidently guesses what went
wrong. It is to build one that can show what it remembers, explain why a
past incident may be relevant, and help an engineer investigate the
present problem.

The bigger idea

The most interesting thing I took away from this project is that an
incident response agent should not be defined only by how well it
analyzes the current request.

It should also be defined by what it can carry forward from previous
work.

Every resolved incident contains experience. Every repeated failure is
an opportunity to make that experience useful again.

Hindsight gave me a practical way to explore that idea: retain useful
operational experience, recall it when a related incident appears, and
keep the human engineer in control of the final decision.

I started with a simple question: why should an incident agent
investigate every failure as if it were the first?

I still think that is the right question to build around.

Publication metadata

Suggested DEV.to tags: ai, agents, python, architecture

Suggested Medium topics: Artificial Intelligence, Software
Engineering, Python, DevOps

Suggested subtitle: Building an incident response agent with
persistent memory using Hindsight, FastAPI, and MongoDB.

Final checks before publishing - Replace the conceptual snippets
with verified repository code, or retain their explicit conceptual
labels. - Verify the API routes against the code. - Add the dashboard,
retain, recall, and architecture screenshots. - Remove all bracketed
screenshot instructions after inserting the images. - Test every
external link and proofread the rendered Markdown. - Do not describe
simulated incident records as verified production incidents.

DEV Community: Kiran V

I Stopped Treating Every Incident as a New Problem

I Stopped Treating Every Incident as a New Problem

The problem: incident history is not the same as memory

The architecture: a familiar application with a memory layer

Incident dashboard

Why I chose Hindsight

Retain and recall: the important lifecycle

Incident records

AI incident diagnosis

The lesson: retrieved memory is evidence, not truth

What the application exposes

System architecture

What I learned building with memory

1. Persistent memory needs a lifecycle

2. Application storage and agent memory solve different problems

3. Recall should influence investigation, not dictate the answer

4. Explicit operations make an agent easier to debug

5. A credible agent must be honest about its capabilities

Where I want to take it next

The bigger idea

Publication metadata