Why I Stopped Writing Post-Mortems in Confluence and Built an SRE Agent with Memory
At 3:14 AM last month, a junior engineer on our team restarted a primary PostgreSQL database to resolve a sudden connection pool spike. What should have been a minor traffic hitch turned into four hours of downtime, dropped checkout transactions, and an emergency rollback because restarting an active pool severed in-flight queries and created a thundering herd.
The worst part? We had solved this exact problem three months earlier. The post-mortem was meticulously documented in our internal wiki—and completely invisible to the engineer on call.
Static post-mortems and runbooks are where operational knowledge goes to die. When production is burning, nobody searches a wiki for historical incident patterns. Stateless AI agents aren’t much better; ask a generic LLM what to do about a saturated connection pool, and it will happily hallucinate "restart the database" with zero awareness that this exact action wrecked your cluster in the past.
To solve this, we built MemoryOps, an incident response agent designed around persistent episodic memory using Vectorize agent memory
. Instead of starting every outage from a blank context window, MemoryOps retains past incidents, recalls verified resolutions, and explicitly remembers which troubleshooting actions failed.
What the System Does and How It Hangs Together
When an alert fires in production, MemoryOps ingests the telemetry, queries a dedicated organizational memory bank, and retrieves historical incident signatures before recommending any remediation.
The architecture comprises three core components:
The Ingestion & Signal Layer: Ingests alerts from Prometheus or Datadog along with deployment metadata (commit hash, version, and service tier) and active error logs.
The Cognitive Memory Tier: Implemented using Hindsight
, an open-source agent memory system developed by Vectorize. Hindsight stores past incidents as episodic memories and exposes retrieval primitives (recall, retain, and reflect).
The Human-in-the-Loop SRE Control Plane: A React and FastAPI application where on-call engineers can audit retrieved historical cases, review confidence scores, approve or reject recommendations, and execute controlled rollbacks.
Mermaid diagram
When an incident resolves, the post-mortem does not go into a static document. It gets stored directly into the Hindsight memory bank, closing the organizational learning loop.
Core Technical Story: Persistent Memory vs. Stateless Hallucination
The central engineering challenge with AI agents in operations is that standard LLM context windows are ephemeral. Once a session ends, the agent forgets everything it experienced. If you try to dump every historical runbook into a system prompt, you hit context window limits, dilute retrieval relevance, and rack up substantial latency.
Vectorize Hindsight addresses this by treating operational knowledge as structured episodic memories rather than brute-force context stuffing. As documented in the Hindsight documentation
, Hindsight allows an agent to index facts, entities, and temporal context into bank partitions.
Here is how our backend integrates with the Hindsight Python SDK to recall prior incidents:
python
from hindsight_client import Hindsight
client = Hindsight(
base_url=settings.HINDSIGHT_API_URL,
api_key=settings.HINDSIGHT_API_KEY
)
def recall_historical_context(service_name: str, error_message: str):
query = f"{service_name} {error_message}"
# Query Hindsight memory bank
recalled = client.recall(
bank_id="memoryops-prod-cluster",
query=query
)
return recalled.results
When an alert fires, MemoryOps retrieves the top matching historical cases and parses both the successful fix and the failed actions.
Negative Learning: Remembering What Failed
Most AI systems only retrieve positive matches. In site reliability engineering, knowing what not to do is often more critical than knowing what to do.
In MemoryOps, whenever a troubleshooting action fails (or when a post-mortem identifies a counter-productive step), we retain that action with an explicit negative disposition flag:
python
def record_failed_action(incident_id: int, action_type: str, reason: str):
content = (
f"[FAILED ACTION ALERT - INCIDENT #{incident_id}]\n"
f"Action: {action_type}\n"
f"Outcome: FAILED / HARMFUL\n"
f"Reason: {reason}\n"
f"Instruction: DO NOT ATTEMPT without explicit verification."
)
client.retain(
bank_id="memoryops-prod-cluster",
content=content,
context=f"negative-action-{incident_id}"
)
When a new incident exhibits similar symptoms, the agent compares candidate actions against this negative memory archive. If a candidate action matches a previously failed action, MemoryOps actively flags it as dangerous and blocks it in the UI.
Concrete Interaction: Before vs. After Memory
To evaluate the system, we tested MemoryOps against a live simulated outage: Incident #104 on our Payment API.
The Outage Symptoms
Service: Payment API
Error: FATAL: remaining connection slots are reserved (PG-53300)
Active Deployment: v3.2.0 (deployed 10 minutes prior)
Metrics: Error rate 73%, p99 latency 8.2s, DB pool 100% saturated
Mode 1: Without Memory (Stateless Baseline)
Running the incident through a stateless LLM produced standard textbook advice:
AI Diagnosis: "Database capacity exceeded under sudden traffic spike."
Recommended Action: "Restart primary database service to clear idle connections."
Risk: High. Restarting during active queue saturation risks dropping thousands of checkout sessions and corrupting connection pools.
Confidence: 45%
Mode 2: With Hindsight Memory
When we enabled Hindsight episodic memory, the agent recalled Incident #27 (91% semantic similarity):
Recalled Case: Incident #27 (21 days ago) on Payment API.
Historical Evidence: v3.1.0 introduced an unclosed session leak in the payment callback handler.
Negative Memory Alert: "Restarting the database was attempted in Incident #27 and FAILED, causing cascading downtime. DO NOT restart the database."
Verified Resolution: "Rollback deployment v3.2.0 to v3.1.9. Resolved previous outage in under 3 minutes."
Confidence: 94%
The engineer approved the rollback with a single click. The deployment rolled back, the connection pool drained within 40 seconds, and error rates dropped to 0.02%.
Lessons Learned
Outcomes matter more than events: Retaining raw telemetry is noise. What matters is retaining the action taken paired with the outcome achieved (success vs. failure).
Negative memory prevents catastrophic repeats: SRE teams often repeat bad troubleshooting steps because past failures aren't searchable. Archiving failed actions directly into the memory bank provided our biggest reliability gain.
Keep humans in the loop for execution: AI agents should recommend, calculate confidence, and cite historical evidence—but an engineer must approve production-impacting rollbacks or restarts.
Episodic memory beats large context windows: Querying an indexed memory bank via Hindsight
gave us sub-second recall latency without burning tokens on hundreds of irrelevant runbooks.
What's Next
We are expanding MemoryOps to use Hindsight's reflect capability across multi-service clusters, synthesizing cross-service dependency graphs to identify architectural failure modes before deployments even reach production.
When organizations give their agents persistent memory, on-call teams stop firefighting in the dark.
Top comments (0)