My On-Call Agent Stopped Suggesting the Fix That Already Failed

Maheen Shaik — Mon, 09 Nov 2026 16:55:49 +0000

Hey everyone 👋,

Remember that time our database pool ran dry, and our AI assistant told us to increase the connection pool? We had already done that months earlier, and it only made things worse: Postgres hit max_connections, and every service slowed down. The real culprit was a nightly reconciliation job holding connections open.

The assistant wasn’t being dumb — it just had no memory of what happened before. Every debugging session started from zero, doomed to repeat the same mistakes.

So I built an on-call agent that remembers, using Hindsight as its memory layer. Here’s what we built together, what worked for us, and the challenges we hit along the way.

What We Built

An engineer pastes an alert into our little UI page. The agent recalls similar past incidents and human corrections, proposes a root cause and a fix, and shows which past memories it relied on. The engineer then marks the answer right or wrong and can log the real cause, the fix that actually worked, and any fix that failed. All of that gets written back into the memory bank.

The tech stack is intentionally simple: FastAPI, Groq for the LLM, and Hindsight for agent memory. One page of UI, with an obvious Memory ON/OFF toggle so we can compare fairly.

The loop is deliberately crisp:

Recall similar incidents and corrections before answering.
Answer, with recalled memories baked into the prompt.
Retain the alert and the verdict, then later the human’s feedback.

Recall Before Every Verdict

All the Hindsight calls live in a single file, app/memory.py. Recall is a one-liner:

resp = self._client.recall(
    bank_id=bank_id, query=query, max_tokens=2048, budget="mid"
)
texts = [getattr(r, "text", str(r)) for r in (resp.results or [])][:limit]

Those memories go straight into the prompt, along with a few rules for using them. The rule I care about most:

- If a memory says a fix FAILED or was rejected, never suggest
  that fix again; say why you are avoiding it.

Memory is only useful if the model is told what to do with it. Without that instruction, recalled text is just background noise the model might ignore.

Retaining Corrections (Not Just Chats)

Early on, my first instinct was to store every alert and every answer. That was a trap: if the agent guesses wrong and you store the guess as fact, you’re teaching it a mistake.

So we built two kinds of retention. After triage, we store the alert and the verdict — explicitly tagged as not yet confirmed by an engineer. After feedback, we store the confirmed outcome:

parts.append(f"CONFIRMED ROOT CAUSE: {actual_cause}")
parts.append(f"FIX THAT WORKED: {actual_fix}")
parts.append(
    f"FIX THAT DID NOT WORK (do not suggest again "
    f"for this kind of alert): {failed_fix}"
)

Feedback goes under its own document_id, so it never overwrites the triage record. Storing failed fixes as first-class negative constraints turned out to be the most useful decision in our project.

Before vs. After: A Real Example

Take a connection-pool alert like HikariPool-1 - Connection is not available, request timed out after 30000ms.

Memory OFF — The answer is the textbook one: "the pool is too small or connections are leaking," so you look at pool settings and raise the maximum. Reasonable, and exactly the advice that failed for us.

Memory ON, after one correction — We logged the real cause: a nightly reconciliation cron with no statement timeout. Terminating those sessions fixed it, and raising the pool did not. When a similar alert comes in next time (from a different service, even different wording), the recalled memories surface. It flags the cron as the likely cause and explicitly avoids the pool-size fix.

Nothing changed except memory — same model, same prompt template, same alert format. Just the ability to remember.

Watching the Learning Curve

I wanted to see this as a curve, not just a single example. We built 40 synthetic incidents across 8 recurring failure patterns, each with a root cause specific to our team (the cron job, a feature flag, a retention setting). These specifics matter — if the cause is guessable from the log line alone, a stateless model does well, and memory shows no benefit.

The demo runs incidents in waves: one per pattern per wave, running each alert twice — once with memory off and once with memory on. After every memory-on run, the "human" confirms the true cause, so Hindsight retains it.

Our results on 24 incidents (3 waves of 8 patterns):

Wave	Memory OFF	Memory ON
1 (cold start)	0/8	0/8
2	0/8	6/8
3	0/8	8/8

Wave 1 scores zero for both because there’s nothing to recall yet. From Wave 2, the memory-enabled agent begins recalling confirmed causes from earlier waves.

The Things That Bit Us

Recall quality depends on what you retain. Vague memories gave vague recalls. Writing each correction as a self-contained paragraph (alert, cause, fix, failed fix) worked far better than short notes.
Retain isn’t instant. Hindsight extracts facts when you retain, which takes seconds. An async option exists, but then recall might not see the newest memory immediately. In our demo, where we retain and recall back‑to‑back, that matters.
Reliability code is not optional. LLMs sometimes return malformed JSON. We validate output with Pydantic, retry three times, fall back to a second model, and finally return a clearly labeled safe default. Hindsight errors become warnings, so the agent keeps working with empty memory instead of crashing.
Keep the memory‑off path honest. In memory‑off mode, we skip recall and retain entirely. Otherwise, the baseline quietly learns too, and the comparison loses its integrity.

The Honest Limitations

The incidents are synthetic and generated from templates — real incident data would be messier. We scored answers with keyword-overlap against the true root cause, which is a crude proxy, not a real accuracy measure. 24 incidents is a small sample. This shows that the idea works, not that it’s ready to run your pager. A real deployment would need authentication, durable stats, and evaluation on your own incident history.

What I’d Tell You If You Build Something Similar

Store outcomes, not chatter. Confirmed causes and failed fixes are worth far more than raw conversation.
Label uncertainty. Mark unconfirmed guesses as unconfirmed so they never become "facts."
Tell the model how to use memory. Recall alone does nothing without explicit rules.
Build a memory‑off switch from day one. It’s the only honest way to know whether memory helps.
Use data where the answer is team‑specific. That’s where memory truly earns its keep.

The code is on GitHub: Memory-Agent. To try the memory layer yourself, start with the Hindsight docs and the Hindsight repo.

Thanks to Code.in for the free coding credits.

DEV Community: Maheen Shaik