My On-Call Agent Stopped Suggesting the Fix That Already Failed

Suhel Engineering — Mon, 09 Nov 2026 15:40:56 +0000

Last time our database pool ran dry, our AI assistant gave the textbook response: increase the connection pool.

We had done exactly that months earlier. It made the outage significantly worse: Postgres slammed into its max_connections ceiling, cascading latency across every connected service. The real culprit was an unoptimized nightly reconciliation job holding idle transactions open.

The AI assistant wasn’t being unintelligent; it simply had no memory. Every new debugging session started from zero context, doomed to repeat past blunders.

To solve this, I built an on-call triage agent that remembers institutional history using Hindsight as its dedicated memory layer. Here is how the system works, what the real-world benchmarks showed, and where its current limitations lie.

System Overview

When an engineer pastes an incoming production alert, the agent runs an active triage loop:

It queries its memory bank for past incidents with similar failure patterns and human corrections.
It generates a diagnosis proposing a root cause and resolution, explicitly citing which memories informed the decision.
The on-call engineer reviews the diagnosis, marks it accurate or inaccurate, and logs the real verified cause, the fix that resolved it, and any fix that failed. All of this feedback is persisted back into memory.

The architecture remains intentionally lean: FastAPI on the backend, Groq for high-speed LLM inference, and Hindsight handling long-term agent memory. The frontend is a single-page interface featuring an explicit Memory ON/OFF toggle to allow rigorous, controlled A/B evaluations.

The core execution loop follows three deliberate stages:

Recall: Retrieve relevant prior incidents and verified human corrections before answering.
Answer: Generate a diagnostic verdict with recalled context injected directly into the system prompt.
Retain: Index the alert and unconfirmed verdict immediately, then append verified human corrections once triage concludes.

Recall Before Every Verdict

All interactions with Hindsight are encapsulated inside app/memory.py. When an alert is received, the recall routine performs a similarity search across the tenant bank using the alert payload as the search query:

python
resp = self._client.recall(
bank_id=bank_id, query=query, max_tokens=2048, budget="mid"
)
texts = [getattr(r, "text", str(r)) for r in (resp.results or [])][:limit]

These recalled historical records are injected directly into the model’s system prompt alongside strict operational constraints. The most critical directive in the system instructions is:

text

If a memory says a fix FAILED or was rejected, never suggest that fix again; say why you are avoiding it.

Memory injection without negative constraints is ineffective. If the prompt does not explicitly instruct the model on how to handle failed remediation attempts, the LLM treats past failures as generic background context and frequently re-recommends them.

Retaining Corrections, Not Just Raw Conversations

A common design pitfall when building memory-augmented agents is storing every conversation turn indiscriminately. If an agent produces a flawed hypothesis and you save that output as ground truth, you permanently poison the agent’s memory bank with bad telemetry.

To prevent this, the storage pipeline separates unverified guesses from confirmed outcomes into two distinct retention phases:

Initial Triage Phase: The incoming alert and initial verdict are stored immediately, but explicitly tagged as unconfirmed by an engineer.
Post-Incident Feedback Phase: Once an engineer verifies the actual resolution, the verified findings are stored under a separate document_id:
python
parts.append(f"CONFIRMED ROOT CAUSE: {actual_cause}")
parts.append(f"FIX THAT WORKED: {actual_fix}")
parts.append(
f"FIX THAT DID NOT WORK (do not suggest again "
f"for this kind of alert): {failed_fix}"
)

Because feedback is stored under an independent document identifier, it never overwrites the historical audit trail of the original triage. Treating failed remediation attempts as first-class negative constraints proved to be the single most impactful architectural decision in this project.

Side-by-Side Comparison: Memory OFF vs. Memory ON

Consider an alert such as:
HikariPool-1 - Connection is not available, request timed out after 30000ms

Memory OFF (Baseline): The model produces standard generalist advice: the connection pool is undersized or connections are leaking, so check pool configurations and increase maximum pool size. This is logically sound in isolation, yet it represents the exact counterproductive remediation that worsened our original outage.
Memory ON (After One Past Correction): In an earlier incident, the engineer logged that the root cause was a nightly reconciliation cron running without a statement timeout, that terminating the offending database sessions resolved the issue, and that scaling up pool sizes caused severe degradation. When an alert with similar characteristics arrives later—even from an entirely different microservice or phrased in different terms—the recalled memories populate the prompt. The agent correctly flags the reconciliation cron as the probable root cause and explicitly advises the operator not to increase pool limits.

Everything else remains identical: identical LLM, identical prompt template, and identical alert style. The only variable altered is memory retention.

Quantifying the Learning Curve

To measure this behavior systematically rather than relying on isolated anecdotes, I evaluated the agent across a benchmark suite of 40 synthetic incidents covering 8 recurring failure patterns.

Each failure pattern was mapped to an internal, team-specific root cause (e.g., an unindexed reconciliation cron, an undocumented feature flag flag-flip, or a silent retention policy mismatch). These private contextual nuances are critical: if a root cause can be inferred solely from generic public documentation or log syntax, a zero-shot model will answer correctly and memory provides no measurable advantage.

The test suite replays incidents in iterative waves (one alert per pattern per wave). Every alert is executed twice under identical conditions: once with memory disabled, and once with memory enabled. Following each memory-enabled execution, the benchmark harness simulates engineer feedback by committing the verified resolution into Hindsight.

Here are the results across 24 incidents (3 successive waves of 8 distinct patterns):

Wave Memory OFF (Baseline) Memory ON
Wave 1 (Cold Start) 0 / 8 0 / 8
Wave 2 0 / 8 6 / 8
Wave 3 0 / 8 8 / 8
Analyzing the Data
Wave 1: Both configurations score 0/8. Because the memory bank is initially cold, neither mode has access to prior institutional incident logs.
Wave 2: The memory-enabled agent recalls confirmed causes recorded during Wave 1, jumping to a 75% accuracy rate (6/8), while the baseline remains at 0/8.
Wave 3: With two waves of institutional context retained, the memory-enabled agent achieves a 100% accuracy rate (8/8), consistently identifying internal root causes and filtering out past invalid fixes.
Engineering Challenges and Pitfalls Encountered
Recall Quality Directly Mirrors Retention Structure: Unstructured or brief notes produced low-relevance semantic matches. Storing corrections as structured, self-contained summaries containing the original alert, confirmed cause, successful resolution, and explicit failed attempts produced significantly higher recall precision.
Fact Extraction Latency: Hindsight parses and extracts entity relationships upon retention, which takes a few seconds. While an asynchronous ingestion mode exists, using async can introduce read-after-write lag where an immediate subsequent query misses the most recent memory. This balance must be handled deliberately in high-frequency evaluation pipelines.
Defensive Output Parsing is Essential: Real-world LLM outputs occasionally produce invalid JSON. The pipeline enforces strict Pydantic validation, exponential backoff retries across three attempts, an automatic fallback to an alternative model, and ultimately returns a gracefully formatted safe fallback. Upstream memory retrieval failures are caught and downgraded to warnings so triage continues smoothly even if the memory bank is temporarily unreachable.
Baseline Integrity Must Be Strictly Maintained: In Memory OFF mode, both recall queries and write/retention events must be bypassed entirely. If baseline runs write feedback to the storage layer, the baseline silently learns, invalidating your comparison benchmark.
Honest Limitations
Synthetic Evaluation Dataset: The incidents evaluated were synthetically generated from templates. Production incident streams contain noisy logs, incomplete stack traces, and concurrent alerts that will test edge cases more aggressively.
Scoring Methodology: Benchmark scoring was calculated via automated keyword-overlap against ground-truth root cause labels. While functional as a directional proxy, keyword overlap is not a replacement for human semantic review.
Sample Size: A 24-incident sample validates the architectural viability of negative-constraint memory, but larger production benchmarks are necessary before delegating production runbooks. Production deployment additionally requires persistent metrics storage, role-based access control, and historical log ingest pipelines.
Key Takeaways for Building Memory-Augmented Agents
Retain Verified Outcomes, Not Ephemeral Dialogue: Storing validated root causes and rejected actions yields far higher signal-to-noise than logging raw conversational transcripts.
Tag Uncertainty Explicitly: Mark unverified agent assertions as tentative immediately so hallucinations do not become trusted institutional memory.
Instruct the Model on How to Act on Recalled Knowledge: Retrieval is only half the battle; provide explicit negative prompts instructing the agent never to recommend failed steps.
Implement an A/B Isolation Switch Early: Building a zero-state toggle into your architecture from day one is the only way to objectively measure whether memory is delivering value.
Focus Memory on Team-Specific Edge Cases: The highest ROI for agent memory lies in organizational, bespoke quirks that general-purpose foundation models cannot know out of the box.
Code & Resources
Repository: Memory-Agent on GitHub (https://github.com/FirdaushAnwer/Memory-Agent)
Memory Framework: Hindsight Documentation & GitHub Repo (https://github.com/vectorize-io/hindsight)
Conceptual Deep-Dive: What is Agent Memory? (https://hindsight.vectorize.io/)

Special thanks to Code.in for providing the coding credits that supported this build.

DEV Community: Suhel Engineering

My On-Call Agent Stopped Suggesting the Fix That Already Failed