OpsRecall AI: Building an AI Incident Response Assistant That Learns From Past Incidents

When an application goes down, engineers often spend valuable time investigating logs, identifying the root cause, and searching for solutions to similar incidents that happened in the past.

Even when a team has already solved a particular issue, that knowledge can get lost in incident reports, chat conversations, or individual experience. The next engineer facing a similar problem may have to start the investigation from scratch.

I wanted to explore how AI and persistent memory could help solve this problem.

That led me to build OpsRecall AI, an AI-powered incident response assistant that helps engineers investigate technical incidents, retain verified resolutions, and recall relevant knowledge when similar problems occur again.

The central idea is simple:

An incident should not just be resolved. It should become useful knowledge for the next incident.

What Is OpsRecall AI?

OpsRecall AI is a web-based incident response assistant designed to support engineers during incident investigation.

It combines an AI-powered analysis workflow with a memory layer inspired by Hindsight's Retain, Recall, and Reflect operations.

The system supports three main capabilities:

Investigate: Analyze incident information and generate possible causes and troubleshooting guidance.

Retain: Store knowledge from an incident after an engineer verifies its resolution.

Recall: Retrieve relevant knowledge from previous incidents to assist with future investigations.

It also includes a Memory Explorer to inspect stored incident knowledge and a reflection feature to synthesize useful insights from memory.

Technology Stack

I used the following technologies to build the application:

Component

Technology

Frontend

React 19, Vite

Backend

Python, FastAPI

AI model

Groq LLM

Memory workflow

Hindsight-inspired Retain, Recall, Reflect

Local metadata and fallback memory

SQLite

The frontend provides the incident management interface, while the FastAPI backend handles the investigation and memory workflows.

Groq is used for AI-powered incident analysis. SQLite supports application metadata and the local memory fallback used in the tested demo environment.

System Architecture

The application follows a frontend-backend architecture.

An engineer submits incident details through the React interface.

The frontend sends the incident information to the FastAPI backend.

The backend processes the incident and uses the configured LLM to assist with analysis.

When an engineer verifies a resolution, the system can retain that knowledge.

When another incident is submitted, the memory workflow can retrieve related knowledge.

The engineer can inspect memory and generate reflections through the interface.

This separation makes it easier to maintain the frontend, backend, AI analysis, and memory components independently.

The Three Core Memory Operations

One of the most interesting parts of this project was exploring how memory can make an AI incident response workflow more useful over time.

1. Retain — Learning From Verified Resolutions

An AI-generated suggestion should not automatically become trusted operational knowledge.

For this reason, OpsRecall AI includes an engineer verification step.

After an incident is investigated, the engineer can review the proposed resolution, confirm whether it worked, and submit the verified resolution for retention.

This helps distinguish between an AI-generated suggestion and a resolution that has actually been verified.

1. Recall — Reusing Previous Knowledge

When a new incident is submitted, the system can search for related information from previously retained incident knowledge.

Instead of treating every incident as completely new, the application can use relevant past resolutions as additional context for investigation.

This is especially useful when different services experience similar underlying problems.

1. Reflect — Finding Patterns

Recall helps retrieve relevant knowledge, while reflection helps synthesize information into broader insights.

OpsRecall AI includes a reflection workflow that allows engineers to explore patterns and lessons from stored incident knowledge.

Together, these operations demonstrate how an AI system can move beyond one-time responses toward a workflow that uses accumulated knowledge.

A Practical Demo: Two Related Incidents

To demonstrate the workflow, I implemented two incident scenarios.

Scenario A: Orders API — Database Connection Pool Exhaustion

The first scenario involves an Orders API experiencing database connection pool exhaustion.

The engineer submits the incident and requests AI investigation. After reviewing the analysis, the engineer verifies the resolution and retains it as incident knowledge.

Scenario B: Checkout API — API Timeout

The second scenario involves a Checkout API experiencing an API timeout.

Although this is a different service, it can have similarities with the earlier database connectivity issue.

When the engineer investigates this incident, the application demonstrates how related information from the earlier incident can be recalled and used as context.

This demonstrates the main idea behind OpsRecall AI: knowledge from one verified incident can help engineers investigate another related incident.

Handling Sensitive Information in Logs

Incident logs can contain sensitive information, including API keys, passwords, JWTs, and database connection strings.

Sending such information directly into an AI workflow can create unnecessary security risks.

I therefore included a log sanitization step to identify and mask sensitive patterns before processing.

The goal is to reduce accidental exposure of credentials while keeping useful diagnostic information available for investigation.

This is an important part of building AI tools for operational environments, where logs may contain more than just technical error messages.

Testing and Validation

During development, I tested the main backend workflows and the frontend build.

The results from the demo validation were:

Backend tests: 11/11 passed

Frontend production build: Passed

Demo verification script: Passed

These checks helped validate the implemented workflows and catch integration issues during development.

The project also includes a local SQLite fallback for memory operations. In the tested demo environment, this fallback was active. Although Hindsight Cloud configuration was added, a successful live Cloud synchronization was not verified in that environment.

This distinction is important: the demo demonstrates the incident memory workflow, but it should not be interpreted as proof of a fully verified cloud deployment.

Challenges and What I Learned

Building OpsRecall AI helped me understand that integrating an LLM into an application is only one part of building a useful AI system.

Some of the key lessons were:

1. AI suggestions need verification.
   An LLM can generate useful troubleshooting guidance, but operational knowledge should not automatically be treated as correct. Human verification is an important safeguard.

2. Memory needs a purpose.
   Simply storing previous conversations or incidents is not enough. The system needs a way to retain useful knowledge and retrieve it when it becomes relevant.

3. Security must be considered early.
   Log sanitization is important when working with real-world operational data.

4. Integration takes careful testing.
   The frontend, backend, LLM, and memory layer each have their own responsibilities. Testing them together is essential.

What's Next?

OpsRecall AI is an evolving project. Some areas I would like to improve include:

Stronger semantic retrieval across a larger incident knowledge base.

Fully verified Hindsight Cloud integration and synchronization.

More incident categories and realistic test scenarios.

Improved observability and incident analytics.

Role-based access and stronger security controls.

Evaluation of retrieval relevance and the quality of AI-generated recommendations.

Conclusion

OpsRecall AI started with a simple question:

What if an incident response assistant could use knowledge from previous verified incidents to help engineers solve future problems?

By combining AI-powered analysis with Retain, Recall, and Reflect memory workflows, I built a prototype that explores this idea.

The most important takeaway for me is that useful AI systems should not only generate answers. They should also help preserve knowledge, make it accessible, and support better decisions over time.

This project was a valuable opportunity to work with LLM integration, backend development, memory workflows, and practical AI application design.

Project: OpsRecall AI

Every Incident Teaches the Next.