The most expensive sentence in incident response is "I think we've seen this before." Someone on the team has seen it. They fixed it in June and wrote half a post-mortem, and the one thing that would save you tonight is buried in that doc: don't raise the connection pool size, that's what took the database down last time.
I built RecallOps to make that sentence cheap. It's an on-call agent that triages an alert by recalling how similar incidents actually went: what the root cause was, which fix worked, and, the part I didn't expect to matter most, which fixes failed. Its memory layer is Hindsight, an open-source agent memory system. This post covers how it hangs together, the design decisions that mattered, and what I'd tell anyone wiring memory into an agent.
RecallOps triage screen with the live memory log
What the system does
The stack I modeled is a typical fintech backend: a Java/Spring payments-api on HikariCP, a Go auth-service, Postgres (ledger-db) behind PgBouncer, Redis, Kafka, and Kubernetes. Every resolved incident becomes a post-mortem with symptoms, error logs, root cause, fix steps, what didn't work, time to resolve, and who fixed it.
The flow has four steps:
Retain. Each post-mortem is written into a Hindsight memory bank.
Recall. A new alert comes in (title plus raw logs). That text becomes the recall query, and Hindsight returns ranked facts from past incidents.
Reason. An LLM reads the alert and the recalled memory and produces a likely root cause, fix steps citing incident IDs, a list of things to avoid, a confidence score, and a time-to-resolve estimate based on how long the matching incidents actually took.
Learn. When the engineer marks the incident resolved, the real root cause and fix are retained back into memory. The next similar alert benefits.
There's also a team view that uses Hindsight's reflect to answer questions across the whole bank, like "what keeps breaking, and what fixes keep working?"
The code is deliberately boring: memory.py holds every Hindsight call in one class, agent.py holds prompting and parsing, and app.py is a Streamlit UI. The interesting parts are in the decisions.
The core decision: one incident, one document
The first design question with any memory system is the unit of memory. Hindsight doesn't store your text as a blob. It extracts facts, entities and timestamps from what you retain, then retrieves those facts. That's what makes recall good, but it means that by default you get back a pile of sentences rather than "incident INC-1003."
On-call engineers think in incidents, not facts. So every retain call pins the post-mortem to a document_id equal to the incident ID, and carries the real incident timestamp, flat metadata, and tags:
c.retain(
bank_id=self.bank_id,
content=incident_to_memory(inc),
context=f"production incident post-mortem for {inc['service']}",
timestamp=_parse_ts(inc["date"]),
document_id=inc["id"],
metadata=incident_metadata(inc),
tags=incident_tags(inc),
)
Three details here earned their place:
timestamp is the incident date, not "now." Hindsight has temporal retrieval, so "the cert outage in June" is a meaningful query. If you retain everything with today's date, you throw that away.
The incident ID is repeated in every section of the post-mortem text ("Root cause of INC-1003: …", "Things that did NOT work during INC-1003: …"). Extracted facts are shorter than the source. Repeating the ID keeps each fact attributable even when it's read alone.
The bank has a mission. I created it with an instruction to keep error signatures, fixes and failed attempts attached to their incident ID. That nudges extraction toward what an SRE needs.
On the way out, every recalled fact carries its document_id and metadata, so regrouping is a dozen lines:
for r in resp.results or []:
doc = (r.document_id or "").strip()
md = dict(r.metadata or {})
inc_id = md.get("incident_id") or doc
raw.append({"text": r.text, "type": r.type, "incident": inc_id or None})
if r.type == "observation" or not inc_id:
if r.text and r.text not in observations:
observations.append(r.text)
continue
...
Facts with no incident behind them are Hindsight observations: patterns it consolidates across memories in the background. I show those separately, because "connection-pool exhaustion has had three different root causes here" is exactly the kind of thing a tired human misses.
The Memory recalled panel: past incidents, why each matched, and the facts behind them
Why "what didn't work" is the real payload
I expected the value to come from recalling fixes. In practice the most useful output is the Avoid list.
Here's a concrete example from the incident bank. Three separate incidents share the same headline symptom, HikariPool-1 - Connection is not available, request timed out after 30000ms:
INC-1003: a batch job left 37 connections "idle in transaction."
INC-1007: a code path leaked connections on an exception.
INC-1012: a migration left an invalid index, so queries fell back to sequential scans and held connections for 40 seconds.
A fourth incident, INC-1005, is the punchline. After INC-1003, someone raised maximumPoolSize from 40 to 80. Weeks later the autoscaler scaled payments-api to 24 replicas, giving 1,920 potential connections against a Postgres max_connections of 500. FATAL: sorry, too many clients already.
When a new HikariPool alert fires, the two answers look like this:
Without memory, the model gives sensible textbook advice: identify slow queries, scale connection limits temporarily, scale application pods, restart connections. Two of those are the exact moves that hurt this team before.
With memory, RecallOps cites INC-1003, INC-1012 and INC-1007 as matches. Its first fix step is the diagnostic that distinguishes them: check pg_stat_activity for idle-in-transaction sessions versus long-running active queries. And it warns:
Raising HikariCP maximumPoolSize. Increases load on the database and leads to swapping or connection exhaustion. INC-1003, INC-1005, INC-1007 Restarting PgBouncer. Drops in-flight transactions and causes a spike in checkout errors. INC-1012
That's the difference between an assistant that is right in general and one that is right here. A generic model can't know your history of bad ideas. Memory can.
Same alert, two answers: generic vs memory-backed
Watching it learn
The learn step is what makes this an agent rather than a search box. After triage there's a "Mark resolved" form, pre-filled with the agent's guess, that the engineer corrects. In one run I recorded a root cause that didn't exist anywhere in memory: a new settlement-report CronJob running analytics queries against the Postgres primary and holding about 40 connections. The fix was to move it to the read replica and give batch jobs their own PgBouncer pool. Saving it as INC-1016 took about five seconds.
I re-ran the identical alert. INC-1016 came back as the top match, labeled as learned this session. The first three fix steps changed to "suspend the offending CronJob," "identify blocking queries," and "point it at the read replica," all citing INC-1016. Even the time-to-resolve estimate switched to "based on INC-1016's resolve time." Nothing was retrained. One retain call changed the next answer.
I also tried an alert that had no precedent: JWT validation failing with "token used before issued" on nodes whose clocks had drifted 61 seconds. Recall still returned something (it always does), but the agent correctly treated those incidents as unrelated and fell back to general knowledge about clock skew. Getting that honest "I haven't seen this" behavior took some work, covered below.
Making the comparison fair
A before/after demo is easy to rig: give the "after" side a better model or a better prompt and call it memory. I wanted the comparison to hold up to scrutiny.
RecallOps can use any OpenAI-compatible LLM, but it can also run on Hindsight alone. reflect uses the LLM built into Hindsight Cloud, and it accepts a JSON response_schema. So when no external LLM key is configured, both sides of the comparison use the same model through the same API. The only difference is the memory bank behind it:
def triage_without_memory(self, alert: dict) -> dict:
if self.engine == "hindsight":
# Same model, but pointed at an EMPTY bank: a fair "no memory" baseline.
f = _alert_fields(alert)
self.memory.ensure_named_bank(BASELINE_BANK, BASELINE_MISSION)
res = self._reflect_json(
REFLECT_BASELINE_QUERY.format(**f), bank_id=BASELINE_BANK, label="reflect·baseline",
context=f"{SYSTEM}\n\nAlert logs:\n{f['logs']}",
)
An intentionally empty bank is the cleanest control I could think of. If the right-hand answer is better, memory made it better.
What was painful
Structured output can only extract what the prose contains. My first version handed my big JSON-instructions prompt to reflect along with a response_schema. The output was valid JSON with one fix step and an empty avoid list. The reason: reflect writes an answer first and then extracts it into your schema, and my query got a short answer. The fix was to ask reflect for an explicit runbook (matched incidents, root cause, five to seven numbered steps with incident IDs, failed remediations, a time-to-resolve estimate) and put the alert logs and recalled memory in context. Same schema, and suddenly six cited steps and four warnings. Ask for what you want extracted.
Models list incidents to say they don't match. For the novel clock-skew alert, the model returned three "matched incidents" whose reasons read "did not involve clock skew." Rendered as cards with a 🎯 icon, that looks like a confident wrong answer. I now filter them out after parsing:
non_match = re.compile(r"\b(did not|does not|doesn't|didn't|no match|not related|unrelated|not relevant)\b", re.I)
matched = [
m for m in _as_list_of_dicts(d.get("matched_incidents"), "id")
if re.search(r"INC-\d{4}", str(m.get("id", ""))) and not non_match.search(str(m.get("why", "")))
]
It's crude, but it's the right layer. Recall is supposed to be generous, and relevance is the reasoning step's job, so the UI has to respect that judgment instead of showing everything recall returned.
No tool calling, on purpose. Function calling is brittle across providers. Every LLM call asks for plain JSON instead, then strips blocks and code fences, pulls out the first balanced {...}, tolerates trailing commas, retries once with "that wasn't valid JSON," and finally degrades to showing the raw answer. Boring, and it hasn't broken on me yet.
Read the error body. My first Hindsight key returned 403 on every bank call. The response body said exactly why: the key was scoped to Hindsight's inference API (embeddings, rerankers), not the Cloud memory API. Thirty seconds with requests saved me an hour of debugging auth headers.
Making memory visible
Memory that works invisibly is hard to trust during an outage. The UI shows every Hindsight call (retain, recall, reflect) in a live sidebar log with latency. In my runs, recall over a few hundred memory units took about 1–2.5 seconds, and retaining a post-mortem took about 5. The memory panel shows the actual recalled facts under each incident, not just a summary. When an engineer sees "INC-1012: terminating idle-in-transaction sessions did not help; the bottleneck was slow active queries," they understand why the agent is steering them where it is.
Lessons I'd reuse
Choose the memory unit your users think in. For on-call that's the incident. A document_id plus metadata lets you recall at fact granularity and present at incident granularity.
Store failures as first-class memory. "What didn't work" was the highest-value field in every post-mortem. Write it into memory explicitly and name it explicitly in your retain mission.
Use real timestamps. Retaining historical data with "now" quietly destroys temporal recall.
Make your baseline honest. Same model and same API with an empty bank is a control that skeptics accept. A different prompt is not.
Recall generously, filter in reasoning, and show your work. Let memory return a lot, let the model decide relevance, and then show the user both the decision and the evidence.
If you want to try this pattern, the Hindsight documentation covers retain, recall and reflect in a few pages, and Vectorize's write-up on what agent memory is and why it matters is a good primer on the concepts. The RecallOps code is on GitHub at Yashaszz/recallops. Point it at your own post-mortems and see what it remembers that you forgot.
Top comments (2)
github.com/vectorize-io/hindsight
github.com/Yashaszz/recallops