Title: Giving On-Call Engineers a Memory: Building On Call Memory with Hindsight

saanvii248 — Tue, 29 Sep 2026 19:17:59 +0000

Article
Production incidents have a strange property: the same ones keep coming back. A database connection pool fills up in March, a different service leaks connections in August, and each time an engineer starts from zero. The knowledge exists somewhere, in a postmortem doc, a Slack thread, or the head of someone who left the company. It just isn't available at 3 a.m. when the pager goes off.
For the Hindsight hackathon, our team built OnCall Memory, an incident response agent for a fictional fintech, Northwind Pay. It analyzes a new alert and stack trace, then recommends a root cause and fix. Its main feature is that it remembers every past incident, what fixed it, and what made things worse.
The problem with stateless assistants
A general-purpose LLM can read a stack trace and suggest that a connection pool is exhausted. That advice is generic. It doesn't know that your team's checkout-service leaks connections from Celery tasks, or that rebooting the RDS instance last spring caused an 85-minute thundering-herd outage. Without memory, an assistant gives the same answer on the twentieth incident as on the first.
Architecture
OnCall Memory is a FastAPI backend with a single-page frontend. When an engineer submits an alert, the backend runs the same analysis twice, side by side:
Without memory: the LLM sees only the alert and logs.
With memory: the backend first queries Hindsight for relevant past incidents and passes them to the LLM as context.
The UI shows both answers next to each other, so the value of memory is visible in seconds. We added a Demo Mode that walks through three scenarios: an initial alert, the same root cause under different symptoms, and a case where the agent warns against a fix that failed before.
How we use Hindsight
Hindsight gives an agent three operations, and we used all three.
Retain. We seeded a memory bank named northwind-oncall with 25 realistic incidents. Each includes a date, affected service, error logs, root cause, fix steps, time to resolve, and whether the fix worked. Hindsight breaks these into smaller memories, so 25 incidents became about 90 entries. Retain is also the learning loop. After an incident, the engineer marks the fix as worked or failed and describes what happened. That text is retained, and the memory counter in the UI goes up. The next similar alert can use it immediately.
Recall. When a new alert arrives, we query the bank with the alert text and logs. Hindsight returns the most relevant memories, which we show in the UI as inspectable chunks with their source incident. Showing them matters. A judge, or a skeptical engineer, can see which past incident drove the recommendation.
Reflect. The Systemic Insights button uses reflect across the whole bank. Instead of matching one alert, it asks what patterns keep causing incidents. In our data it found recurring database connection exhaustion, Redis memory problems from missing TTLs, Kafka consumer rebalance loops, and failures from automated infrastructure changes. It also proposed permanent fixes such as connection proxies, TTL enforcement, and pre-deploy validation.
What memory changes
The difference shows up over several interactions. In the first, the agent gives a reasonable diagnosis backed by similar incidents. In the second, the symptoms differ, but recall links the alert to the same underlying cause. By the third, the agent has learned from a failure and tells the engineer not to reboot the database, because that made things worse last time. The recommendation improves because the agent accumulated experience, not because the prompt got better.
Design choices
Visible memory. Recalled chunks are shown, not hidden, because trust in an incident tool depends on evidence.
Failed fixes count. Remembering what didn't work is as valuable as remembering what did.
Realistic data. Incident IDs, log lines and numbers look like real ones, which makes the demo believable.
Where it could go
A production version could ingest PagerDuty alerts and postmortems automatically, run one memory bank per team, and track which fixes worked over time.
OnCall Memory shows that the gap between a generic assistant and a useful one is often memory. Hindsight made adding it a matter of three well-designed operations.

Code:https://github.com/Divyasri03-ux/oncall-memory

OnCall Memory:Building an Incident Response Agent That Learns From Production

saanvii248 — Tue, 29 Sep 2026 18:39:28 +0000

OnCall Memory: Building an Incident Response Agent That Learns From Production History

Production incidents are rarely completely new.

A database connection pool can become exhausted again. A deployment configuration can break a service again. A payment provider can become rate-limited again. The symptoms may change, but the underlying patterns often repeat.

The problem is that a typical AI assistant starts each incident with very little knowledge of an organization's previous experiences.

It can analyze the logs in front of it and provide technically reasonable suggestions, but it does not automatically know what happened during the last incident, which fix worked, or which attempted solution failed.

That is the problem I wanted to address with OnCall Memory, an incident-response website designed around persistent AI memory.

The Idea:

OnCall Memory is built for a fictional fintech company called Northwind Pay.
When an engineer receives a production alert, the website provides two perspectives:

->Without Memory a response generated from the current incident.

->With Hindsight Memory a response generated after retrieving relevant historical incidents from the organization's memory.

This side-by-side comparison is the central idea of the website.

The goal isn't simply to make an AI response longer. It is to give the model access to something it normally doesn't have: the team's previous incident experience.

Building the incident history

The project contains a collection of realistic Northwind Pay production incidents.

The incident dataset includes situations such as PostgreSQL connection pool exhaustion, Redis eviction, expired TLS certificates, bad deployment configuration, Kafka consumer lag, OOMKilled pods, and third-party payment API rate limiting.

Each incident contains information such as the symptoms, logs, root cause, resolution steps, outcome, and whether attempted fixes worked.

This is important because a useful memory isn't just:

"There was a database incident."

It should capture the experience surrounding that incident.

What happened?

Why did it happen?

What was tried?

What actually fixed it?

Did the first solution fail?

That information becomes useful context for future incidents.

Hindsight as the memory layer

Hindsight is at the center of the architecture.

The website uses Hindsight to perform three important operations: retain, recall, and reflect.

When useful incident information is available, it can be retained as memory.

When a new incident arrives, the system recalls relevant historical incidents before generating the memory-enhanced diagnosis.

Finally, the website can use reflection to look across the accumulated incident history and ask a broader question:

What patterns keep causing our incidents and what should we fix permanently?

This moves the system beyond responding to individual alerts and toward learning from patterns across incidents.

The architecture

The website uses a lightweight architecture.

The frontend is a single-page HTML, CSS, and JavaScript dashboard.

The backend is implemented with FastAPI and handles the incident analysis workflow, memory operations, and language-model requests.

Groq provides the language-model layer, while Hindsight provides persistent memory.

The project also separates configuration from application code. API credentials are kept locally in environment variables rather than being hardcoded into the source code, while .env.example provides the required configuration structure.

Memory changes the workflow

Suppose a new payment failure arrives.

Without historical context, an AI assistant might suggest checking the payment provider, network connectivity, credentials, rate limits, retries, and application logs.

Those are reasonable suggestions, but the engineer still has to determine which one matches the organization's previous experience.

With Hindsight memory, the workflow becomes different.

The system first looks for similar historical incidents.

If a previous payment incident involved third-party API rate limiting, for example, that historical information can become part of the diagnosis.

The engineer can also see the recalled incidents rather than receiving a completely opaque recommendation.

That visibility was an important part of the design.

Learning doesn't stop after the first answer

OnCall Memory also includes a feedback loop.

After applying a suggested fix, the engineer can indicate whether the fix worked or failed and provide the actual solution.

That information can then become another memory.

The intended loop is:

Incident → Recall → Diagnosis → Fix → Feedback → Retain → Better future diagnosis

This creates a system where future incidents can benefit from experiences that were not available when the original system was built.

The current Hindsight memory bank contains 139 memories.

What I learned

One of the biggest lessons from building the website was that memory quality matters as much as memory quantity.

Adding more memories does not automatically create a better assistant.

A useful incident memory needs meaningful context: symptoms, root cause, attempted fixes, final resolution, and outcome.

I also learned that memory should be visible to the user.

If an AI gives a very specific operational recommendation, an engineer should have some way to understand why that information was relevant.

Displaying recalled incidents makes the memory layer much easier to inspect.

Limitations

OnCall Memory currently uses a fictional Northwind Pay incident history rather than real production data.

That means the system demonstrates the workflow but should not be interpreted as a replacement for an organization's actual incident-management practices.

The quality of the recommendations also depends on the quality of the stored memories. Poor or incomplete incident records can lead to less useful historical context.

Final thoughts

The central idea behind OnCall Memory is simple:

An incident-response assistant should not forget what happened yesterday.

Language models are good at reasoning about the information they receive. Persistent memory adds another dimension: the ability to use accumulated organizational experience.

By combining Groq for reasoning with Hindsight for persistent memory, OnCall Memory turns incident history into a resource that can be recalled during future incidents and analyzed for recurring patterns.

The result is not an AI replacing an on-call engineer.

It is an AI assistant that can increasingly say:
"We've seen something like this before."

DEV Community: saanvii248

Title: Giving On-Call Engineers a Memory: Building On Call Memory with Hindsight

OnCall Memory:Building an Incident Response Agent That Learns From Production

What I learned