Incident Response Agent — Intelligent Incident Analysis and Response Assistance

Arooba Hasan — Tue, 29 Sep 2026 15:32:23 +0000

When an incident happens at 3 a.m., having historical knowledge is valuable—but only if the engineer can understand why that history is relevant and whether it is still trustworthy.

That was one of the ideas that stood out to me while examining our Incident Response Agent. The system does more than store previous incidents and search through them. It tries to turn incident history into practical guidance while making the reasoning behind that guidance visible.

The most important lesson for me was that an incident-response assistant should not behave as if historical data is automatically correct.

A previous runbook may have worked under very different circumstances. An incident may look similar because two services share the same error signature, while the underlying cause is different. A runbook that worked six months ago may also be less trustworthy today because infrastructure and configuration change.

Our implementation therefore treats historical memory as evidence—not as an unquestionable answer.

The problem: historical incidents are useful, but history gets stale

Incident response often depends on institutional knowledge.

An engineer sees an error and asks questions such as:

Have we seen this before?
What happened last time?
Which runbook did we use?
Did that runbook actually work?
Was the previous incident really comparable to this one?

A basic incident database can answer the first few questions by providing search results. But a list of historical incidents does not necessarily tell the responder which precedent deserves attention.

Our project approaches the problem by creating an incident fingerprint.

The similarity engine looks at five dimensions:

WEIGHTS = {
"text_similarity": 0.40,
"service_match": 0.20,
"error_signature_match": 0.20,
"severity_match": 0.10,
"recency": 0.10,
}

Instead of producing only one unexplained number, the system keeps these dimensions separate.

That distinction matters during an incident. A responder can see that a previous incident matched because it involved the same service and error signature, rather than being told that two incidents are simply “87% similar.”

Making similarity explainable

The text component uses TF-IDF and cosine similarity over information such as the incident title, description, error signature, service, and root-cause summaries.

It is deliberately local and deterministic:

vectorizer = TfidfVectorizer(
stop_words="english",
max_features=2000
)
tfidf = vectorizer.fit_transform(corpus)
text_sims = cosine_similarity(
query_vec,
doc_vecs
).flatten()

The resulting fingerprint contains:

{
"text_similarity": ...,
"service_match": ...,
"error_signature_match": ...,
"severity_match": ...,
"recency": ...
}

The system then combines those values using explicit weights.

This design has an important reliability advantage: the recommendation can be inspected.

For example, a result can be explained through conditions such as:

same service, identical error signature, same severity class

That is much easier for an engineer to challenge than an opaque similarity score.

If the recommendation looks wrong, the responder has some information about why it appeared.

But similarity is not enough

Finding a similar incident is only half of the problem.

Suppose a runbook was used during that historical incident. Should we automatically recommend it?

Our project deliberately avoids treating a simple success percentage as the complete answer.

Instead, each runbook starts with an Elo-style rating of 1200.

The unusual part is that the “opponent” is the incident severity.

SEVERITY_DIFFICULTY = {
"SEV1": 1600,
"SEV2": 1400,
"SEV3": 1200,
"SEV4": 1000,
}

A successful runbook on a SEV1 incident therefore provides stronger evidence than a successful runbook on a SEV4 incident.

Likewise, failing against a routine SEV4 incident costs more rating than failing against a difficult SEV1 incident.

The rating update is based on the standard Elo expectation formula:

expected = expected_score(runbook_rating, opponent)
new_rating = runbook_rating + K_FACTOR * (actual - expected)

This is not intended to prove that a runbook is objectively correct. It is a way of representing accumulated evidence about how the runbook has behaved in incidents of different severity.

Historical success also has a shelf life

One of the most useful reliability ideas in the project is rating decay.

A runbook can have a strong historical rating and still be stale.

Infrastructure changes. Dependencies change. Configuration changes. Teams change operational procedures.

The implementation therefore pulls an unused rating back toward the 1200 baseline:

decay_factor = 0.5 ** (
days_since_last_use / DECAY_HALF_LIFE_DAYS
)
return round(
STARTING_RATING
+ (rating - STARTING_RATING) * decay_factor,
1
)

The half-life is 120 days.

That means the system does not interpret an old successful result as permanent proof.

This is exactly the kind of behavior I think an incident-response system needs. Historical memory should become useful knowledge, but knowledge should remain subject to validation.

Before and after: what changes for the responder?

Consider a new payments-api incident reporting 502 upstream timeouts.

Without the matching and ranking workflow, an engineer might search incident history manually, find several payment incidents, open them individually, and decide which runbook seems relevant.

With the implemented workflow, the system searches resolved incidents and ranks the closest precedents.

The project includes a test specifically checking that an incident with the same service and error signature is ranked appropriately:

assert top_incident.service == "payments-api"
assert fingerprint["service_match"] == 1.0
assert fingerprint["error_signature_match"] == 1.0

The responder therefore receives more than a historical incident title. They receive the similarity fingerprint and the runbooks associated with that historical response.

But the responder still makes the final decision.

That distinction is important.

The system learns from feedback—but does not silently rewrite history

When a runbook is used, the system accepts three outcomes:

worked
partial
failed

The feedback endpoint records the usage and updates the runbook’s Elo rating.

new_rating, delta = elo.update_elo(
rb.elo_rating,
severity,
payload.outcome
)

The previous and new ratings are also stored with the runbook usage.

This creates a feedback loop:

incident → historical match → runbook recommendation → human action → outcome → updated trust

The next recommendation can therefore benefit from previous operational experience.

However, this feedback mechanism is still only as good as the evidence entered into the system. A wrong classification or an inaccurately recorded outcome can influence future recommendations.

That is another reason not to treat the rating as an absolute truth.

Postmortems are deliberately drafts

The same principle appears in postmortem generation.

The project does not call an external LLM to invent a postmortem. Instead, postmortem_gen.py constructs a deterministic draft from the incident’s structured data.

It uses recorded root causes, the timeline, severity, error signature, and MTTR.

For example, it can identify the first detection step and the first mitigating action from the timeline.

detection_steps = [
s for s in steps
if s.step_type == "detection"
]
fix_steps = [
s for s in steps
if s.step_type in ("fix", "mitigation", "resolution")
]

The project explicitly treats the result as a draft.

That is a good reliability boundary. A generated postmortem can organize information and save time, but it cannot know everything a human incident review may need to consider—such as customer impact, organizational context, or the appropriate blameless framing.

The implementation itself makes this limitation clear.

What the tests tell us

The repository contains 17 automated tests, and I think they are an important part of the project’s story.

The tests cover the similarity engine, Elo calculations, and API lifecycle.

For example, the Elo tests verify that:

assert delta_hard > delta_easy > 0

for successful runbooks against different incident severities.

They also verify that failure against an easier incident costs more:

assert delta_easy < delta_hard < 0

The similarity tests verify that the fingerprint contains all five dimensions and that results are ordered by score.

The full API lifecycle is also tested.

When I ran the repository’s test suite, all 17 tests passed. The test run did produce Python deprecation warnings around datetime.utcnow(), which is a useful maintenance issue to address rather than something to hide.

Where the system should not be trusted

The project documents several limitations, and they are important.

First, TF-IDF similarity is lexical.

If two engineers describe the same root cause using completely different terminology, the system may not recognize the relationship as strongly as a semantic embedding system might.

Second, the project uses a single SQLite database.

That is appropriate for the demonstrated application, but the README itself identifies PostgreSQL as a future direction before the system holds a much larger organization’s incident history.

Third, postmortems are deterministic drafts rather than complete incident reviews.

Finally, the runbook rating is evidence, not proof. A “Battle-tested” label does not mean a runbook is guaranteed to work during the next incident.

One important implementation boundary

There is also a distinction worth making clear about memory technology.

The supplied repository does not contain an integration with Hindsight. There are no Hindsight dependencies or retain/recall calls in this version.

Instead, the project’s institutional memory is implemented through its own SQLite data model, incident history, TF-IDF matching, runbook usage records, Elo ratings, and postmortem data.

That does not make the memory mechanism uninteresting. In fact, it makes the reliability lesson easier to see: regardless of the underlying memory technology, historical information needs context, explainability, freshness, and human validation.

Lessons learned

The biggest lesson from this project is that an incident-response assistant should not try to eliminate human judgment.

It should make human judgment faster and better informed.

For me, the most valuable engineering choices were therefore not simply about finding similar incidents. They were the safeguards around that information:

expose why an incident matched;
distinguish different dimensions of similarity;
track whether recommended runbooks actually worked;
account for incident severity when updating trust;
decay stale ratings;
generate postmortems from recorded evidence;
keep generated postmortems as drafts;
test the ranking and feedback logic;
and document the boundaries of the system.

That approach produces something more useful than an assistant that confidently gives an answer.

It produces an assistant that can show its evidence—and gives the engineer enough context to decide whether that evidence still applies.

Conclusion

Incident history becomes valuable when it can influence the next response without becoming a source of blind confidence.

Our Incident Response Agent takes a practical approach: store structured incident history, find comparable incidents, explain the match, rank runbooks using outcome history, reduce confidence in stale knowledge, and turn the response timeline into a reviewable postmortem draft.

The system is not production-ready infrastructure, and it does not prove that its recommendations are correct. Its value is in creating a structured feedback loop around incident knowledge.

That is the reliability lesson I would carry into a larger system: memory should inform the responder, not replace the responder.

DEV Community: Arooba Hasan

Incident Response Agent — Intelligent Incident Analysis and Response Assistance