How OpsMemory AI Turns Production Incidents into Persistent Engineering Experience

# How OpsMemory AI Turns Production Incidents into Persistent Engineering Experience

1. Introduction

Every engineering team that runs something in production accumulates hard-won operational knowledge. Almost none of it is stored anywhere it can be used at the moment it matters — three in the morning, alert firing, symptoms that feel familiar but not familiar enough.

OpsMemory AI is an incident-response assistant built around a single idea: an incident is not finished when service is restored. It is finished when the experience of resolving it is retained somewhere the next incident can reach it. It combines Hindsight for persistent experiential memory with Groq for reasoning, and closes the loop with engineer feedback.

2. The problem

Incidents repeat, in variations. The connection pool exhausts again under a different traffic pattern. Authentication fails again, this time from a different expired credential. The symptoms rhyme; the specifics differ just enough that a text search for the old ticket returns nothing useful.

What makes this expensive is not the diagnosis. It is the repeated dead end — the plausible fix that someone already tried six months ago, that already didn't work, that nobody wrote down because it didn't work.

3. Why incident knowledge gets lost

• Postmortems record what happened, not what was tried and abandoned.
• Ticket search is lexical; incidents are described in whatever words the reporter used that day.
• The person with the context changes teams.
• Chat threads are where the real reasoning happens, and chat threads are unsearchable in practice.

A general-purpose LLM does not fix this either. It knows a great deal about distributed systems and nothing about your last outage. Ask it about connection pool exhaustion and it will give you a competent textbook answer that ignores the fact that your team already ruled out the obvious cause last quarter.

4. OpsMemory AI

The system does four things:

1. Recall related incidents and engineer-verified outcomes from persistent memory.
2. Reason over the current incident with that evidence in the prompt.
3. Recommend a structured analysis: root cause, confidence, evidence, failed approaches, actions.
4. Retain the engineer's verified outcome once the incident is resolved.

Step four is what makes step one better over time.

5. Architecture

Next.js frontend
      ↓
FastAPI backend
      ↓
Incident Agent ──► Hindsight (recall)
      │            Groq (reason)
      ▼
Structured analysis ──► Engineer feedback ──► Hindsight (retain) ──► future incidents

FastAPI with Pydantic contracts, a Next.js App Router frontend, and two external services. The incident agent is the only component that talks to both.

6. Hindsight Retain

Two kinds of document are stored. Historical incidents are seeded with a stable structure — problem, root cause, failed approach, successful resolution. Engineer feedback is retained after resolution, tagged incident-feedback plus the outcome.

Stable document_id values with update_mode="replace" make writes idempotent, so re-seeding updates rather than accumulating duplicates.

The critical field is the failed approach. An outcome recorded as incorrect is retained as evidence against repeating that recommendation — the one thing postmortems systematically omit.

7. Hindsight Recall

Every analysis performs two recalls: one unfiltered for related incidents, one tag-filtered for engineer-verified feedback. They are separate on purpose — a verified outcome is worth more than general incident text, and merging them lets the general text crowd it out.

Hindsight returns extracted observations rather than the raw documents. A stored feedback record comes back as distilled facts like "Scaling the consumer group did not resolve the lag." This is useful, and it shaped the design: matching had to key on distinctive facts rather than incident IDs, because the IDs usually aren't in the extracted text.

8. Groq reasoning

Groq runs openai/gpt-oss-120b at temperature=0.2 with response_format={"type": "json_object"}, returning a schema the application validates with Pydantic. If the JSON is invalid, the agent degrades to a low-confidence analysis that still carries the real evidence, rather than raising.

The prompt is explicit: use only the supplied evidence for historical claims, don't invent incidents, lower confidence when evidence is absent.

9. Feedback loop

The engineer selects an outcome — correct, partially_correct, incorrect, unknown — and supplies the actual root cause and resolution. That becomes a memory. No weights change. Nothing is fine-tuned. The next incident simply has more to recall.

10. Before/after demonstration

This is the claim worth being careful about, because it is the easy one to fake.

The first version of the demo was dishonest without meaning to be. It analyzed an Authentication API incident "before learning", taught the system, then analyzed a similar one "after". But the seed data already contained that exact scenario — expired JWT signing key, restart failed, rotation worked. The "before" analysis was already scoring 95% confidence with the correct answer. The contrast being demonstrated was not real.

Making it honest required isolation. The demo now creates a memory bank for that run alone:

1. Create an empty bank.
2. Analyze incident one. Assert that zero historical evidence was recalled — if memory isn't provably cold, fail rather than show a contrast that can't be justified.
3. Retain the engineer's verified outcome, including the approach that failed.
4. Analyze a second incident on the same service, described in different words.
5. Assert that the recalled evidence matches the facts just retained.
6. Delete the bank.

A shared bank cannot give that guarantee. Once a scenario has been demonstrated once, its memory persists, and every later run's "before" already knows the answer — semantic recall matches the reworded symptoms even under a different service name. That's not a flaw in the memory layer; it's the memory layer working. But it means a demonstration of cold-start behavior needs its own bank.

A representative run:

	Before	After
Historical evidence	0 items	1 item
Likely root cause	"Schema ID 4711 missing from the registry" (a guess from the logs)	"Consumer group pinned to an outdated schema version" (the verified cause)
Confidence	80%	90%
Recommended fix	"Register the missing schema"	"Remove the version pin, redeploy, validate"

Same model, same code path. The only difference is that the experience existed.

11. Evaluation

Measured on every run of run_evaluation.py: Recall@3 across eight semantic retrieval cases, feedback recall, before/after learning, health, and latency. Current numbers, including run-to-run variance, are in evaluation.md — measured, not estimated.

12. Challenges

Recall returns observations, not documents. Assertions written against incident IDs passed early and then began failing as the bank filled up, because the IDs stopped appearing in extracted text. Matching had to move to distinctive facts.

Unrelated memories arriving as "evidence." Semantic recall is generous. Without filtering, an unrelated but broadly sensible observation would be presented to an engineer as historical evidence for their incident. A lexical overlap filter (at least two meaningful shared terms) now gates it — conservative, and deliberately so.

A demo that proved nothing. Covered above. The failure mode was subtle: everything ran, output looked impressive, and the comparison was meaningless.

Import-time credential checks. The Groq module raised on import if the key was missing, which took down /docs and /health on a misconfigured deployment — endpoints that should still answer. Lazy client construction fixed it.

13. Lessons learned

• A demo that can't fail isn't evidence. The strongest change was making the demo assert its own premise and refuse to print a contrast it couldn't prove.
• Retrieval quality is a filtering problem as much as a search problem. Getting relevant memories back was easy; refusing to show irrelevant ones as evidence was the work.
• Keep generated text out of factual fields. Evidence is attached by the application from what memory returned. A model that hallucinates a prior incident cannot get it into that list.
• Non-determinism must be reported, not smoothed. Recall@3 varies between runs. Publishing a range is honest; publishing the best run is not.

14. Limitations

• Incident records are in-memory and reset on restart; only Hindsight persists.
• Evidence filtering is lexical, not semantic, so it can drop a relevant memory phrased differently.
• No authentication or multi-tenancy.
• Most tests call live external services and assert against seeded data.
• No automated frontend tests.
• Recalled observations rarely carry incident IDs, so evidence can't be linked back to a source ticket.

15. Future improvements

Durable incident storage; per-team memory banks with authentication; alert-source integrations so incidents open themselves; semantic reranking to replace the lexical filter; a larger labeled evaluation set tracked over time.

16. Conclusion

The interesting thing about OpsMemory AI is not that it uses an LLM to analyze incidents. It is that the analysis gets better at your systems specifically, from outcomes your engineers verified, without anyone retraining anything.

The mechanism is persistent experiential memory and feedback-driven recall — memory-augmented reasoning, not fine-tuning. That distinction matters practically: memory is inspectable, correctable, deletable, and attributable. When OpsMemory says a prior approach failed, there is a stored record saying so, written by an engineer who was there.

That is what turns an incident from a cost into an asset.