Last time our database pool ran dry, an AI assistant told me to increase the connection pool. We had done exactly that months earlier. It made things worse: Postgres hit max_connections and every service got slower. The real culprit was a nightly reconciliation job holding connections open.
The assistant wasn't being dumb. It just had no way to know. Every session started from zero.
So I built an on-call agent that remembers, using Hindsight as its memory layer. This is what I built, what worked, and where it falls short.
What it does
An engineer pastes an alert. The agent recalls similar past incidents and corrections, proposes a root cause and a fix, and shows which memories it relied on. The engineer then marks the answer right or wrong and can enter the real cause, the fix that worked, and any fix that failed. All of that goes back into memory.
The stack is small: FastAPI, Groq for the LLM, and Hindsight for agent memory. One page of UI, with a Memory ON/OFF toggle so I can compare honestly.
The loop is three steps:
- Recall similar incidents and corrections before answering.
- Answer, with the recalled memories in the prompt.
- Retain the alert, then later the engineer's correction.
Recall before every verdict
All Hindsight calls live in one file, app/memory.py. Recall is a single call, with the alert text as the query:
resp = self._client.recall(
bank_id=bank_id, query=query, max_tokens=2048, budget="mid"
)
texts = [getattr(r, "text", str(r)) for r in (resp.results or [])][:limit]
Those memories go straight into the prompt, along with rules for using them. The rule I care about most is this one:
- If a memory says a fix FAILED or was rejected, never suggest
that fix again; say why you are avoiding it.
Memory is only useful if the model is told what to do with it. Without that instruction, recalled text is just extra context the model may ignore.
Retaining corrections, not just conversations
The first design question was what to store. My first instinct was to save every alert and every answer. That is a trap: if the agent guesses wrong and you store the guess as fact, you have taught it a mistake.
So there are two kinds of retained memory. After triage, I store the alert and the verdict, explicitly labelled as not yet confirmed by an engineer. After feedback, I store the confirmed outcome:
parts.append(f"CONFIRMED ROOT CAUSE: {actual_cause}")
parts.append(f"FIX THAT WORKED: {actual_fix}")
parts.append(
f"FIX THAT DID NOT WORK (do not suggest again "
f"for this kind of alert): {failed_fix}"
)
The feedback goes in under its own document_id, so it never overwrites the triage record. Storing failed fixes as first-class facts turned out to be the most useful decision in the project.
Before and after
Take a connection-pool alert like HikariPool-1 - Connection is not available, request timed out after 30000ms.
Memory off. The answer is the textbook one: the pool is probably too small or something is leaking connections, so look at pool settings and consider raising the maximum. It's reasonable, and it is exactly the advice that failed for us.
Memory on, after one correction. I told the agent the real cause was the nightly reconciliation cron with no statement timeout, that terminating those sessions fixed it, and that raising the pool did not. The next time a similar alert arrives, from a different service and with different wording, the recalled memories show up in the verdict. It names the cron as the likely cause and explicitly avoids the pool-size fix.
Nothing changed except memory. Same model, same prompt template, same alert style.
Making the learning curve visible
I wanted to see this as a curve, not just one example. The test data is 40 synthetic incidents across 8 recurring failure patterns, each with a root cause specific to this team (the cron job, a feature flag, a retention setting). Those specifics matter. If the true cause is guessable from the log line alone, a stateless model scores well and memory shows no benefit.
The demo replays incidents in waves, one per pattern per wave, running each alert twice: once with memory off and once with memory on. After every memory-on run the "engineer" confirms the true cause, so Hindsight retains it.
My run on 24 incidents (3 waves of 8 patterns):
| Wave | Memory off | Memory on |
|---|---|---|
| 1 (cold start) | 0/8 | 0/8 |
| 2 | 0/8 | 6/8 |
| 3 | 0/8 | 8/8 |
Wave 1 is zero for both, which is expected: there is nothing to recall yet. From wave 2 the memory-on agent recalls the confirmed causes from earlier waves.
Things that bit me
- Recall quality depends on what you retain. Vague memories gave vague recalls. Writing each correction as a self-contained paragraph (alert, cause, fix, failed fix) worked much better than short notes.
- Retain is not instant. Hindsight extracts facts when you retain, which takes seconds. There is an async option, but then recall may not see the newest memory immediately. That matters in a demo where you retain and recall back to back.
- Reliability code is not optional. LLMs sometimes return malformed JSON. I validate output with pydantic, retry three times, fall back to a second model, and finally return a clearly labelled safe default. Hindsight errors become warnings, so the agent keeps working with empty memory instead of crashing.
- Keep the memory-off path honest. In memory-off mode I skip recall and retain entirely. Otherwise the baseline quietly learns too and the comparison means nothing.
The honest limitations
The incidents are synthetic and generated from templates, so real incident data will be messier. I scored answers with a keyword-overlap check against the true root cause, which is a crude proxy and not a real accuracy measure. 24 incidents is a small sample. This shows that the idea works, not that it is ready to run your pager. A real deployment would need authentication, durable stats, and evaluation on your own incident history.
What I'd tell you if you build something similar
- Store outcomes, not chatter. Confirmed causes and failed fixes are worth far more than raw conversation.
- Label uncertainty. Mark unconfirmed guesses as unconfirmed so they never become "facts".
- Tell the model how to use memory. Recall alone does nothing without explicit rules.
- Build a memory-off switch from day one. It's the only honest way to know whether memory helps.
- Use data where the answer is team-specific. That's where memory earns its keep.
The code is on GitHub: Memory-Agent. To try the memory layer yourself, start with the Hindsight docs and the Hindsight repo.
Thanks to Code.in for the free coding credits.
Top comments (0)