Most postmortems are write-only documents. An on-call engineer spends four hours debugging a cascading failure at 3:00 AM, writes up a detailed timeline, lists three things that failed before finding the fix that worked, and files the ticket away into an archive where nobody ever reads it again.
Two months later, under similar traffic conditions, another engineer gets paged for the exact same symptom. And what do they do? They repeat the exact same failed fix that made things worse the first time around.
When we set out to build RecallOps—an operational intelligence layer for Kartly, an e-commerce platform processing checkout transactions across six distributed microservices—we wanted to break this loop. We did not want another dashboard that dumps thousands of raw log lines onto an engineer’s screen. We wanted an AI incident response system that actually remembers what happened in previous outages: what alert fired, what recent deployment preceded it, what the on-call engineer tried that failed, and what finally restored service.
To make this work, we had to solve a fundamental architectural challenge: relational databases are built for operational state, not episodic reasoning, while raw vector retrieval over unstructured logs produces noisy, ungrounded hallucinations.
Here is how we designed a dual-tier architecture combining PostgreSQL with Hindsight on GitHub, how we structured operational memory, and what we learned about bridging relational data and long-term agent recall.
What the System Does and How It Hangs Together
At its core, RecallOps sits between our observability stack and our incident response workflow. The infrastructure runs six core microservices: checkout-api, payments-svc, inventory-svc, search-svc, notification-svc, and delivery-eta-svc.
Our operational data pipeline separates concerns across two purpose-built storage engines:
+-------------------------------------------------------------------------+
| PostgreSQL Relational Store |
| |
| [engineers] ---> [services] ---> [alerts] ---> [incidents] |
| | | | |
| v v v |
| [deploys] <---------+ +-------+-------+ |
| | | |
| v v |
| [log_lines] [fix_attempts] |
| (Raw Logs) (worked/failed) |
+--------------------------------------------|----------------------------+
|
Distillation Pipeline | (Extract causal episodes;
(Never send raw logs!) | exclude raw noise)
v
+-------------------------------------------------------------------------+
| Hindsight Cloud Episodic Memory Bank |
| ('kartly-sre') |
| - Temporal Context & Timestamps |
| - Negative Memory (What failed and why) |
| - Successful Remediations & Causal Ground Truth |
+-------------------------------------------------------------------------+
PostgreSQL (The Operational Source of Truth):
Postgres maintains relational integrity: staff engineering ownership, 400 deployments over six months, threshold alerts, 110 incidents (INC-0001toINC-0110), 3,500 raw log lines, and fix attempts strictly categorized asworked,failed, orpartial.Hindsight (The Episodic Agent Memory Engine):
Postgres cannot perform semantic reasoning over past human actions. If an alert fires fordb_connection_pool_active_ratio > 0.90, querying Postgres for exact text matches fails when pod names or stack frames differ. We pipe distilled postmortems into Hindsight, a dedicated long-term memory engine for AI agents that organizes memories into managed banks with temporal reasoning and entity graphs.
The Core Technical Problem: Naive RAG Fails in Operations
When engineers first experiment with LLMs for incident management, the standard approach is naive Retrieval-Augmented Generation (RAG): chunk your application logs, compute embeddings, store them in a vector database, and retrieve nearest chunks when an alert fires.
In production SRE, this approach falls apart immediately:
- Raw Logs Drown Causal Signals: A single incident generates dozens of log lines across HikariCP, PostgreSQL, and Nginx. Most are redundant timeout errors. Vector search on "connection timed out" matches every connection failure in company history regardless of whether the root cause was an expired mTLS cert, a saturated pool, or a downstream gateway outage.
- Standard Embeddings Ignore Outcomes: Cosine similarity cannot distinguish between an action that fixed an outage and an action that crashed the database. If an engineer restarted application pods and exacerbated connection exhaustion, a naive vector search sees "restarted application pods" right next to the incident description and suggests it as a remedy.
- Loss of Temporal Sequence: Incidents are chronological narratives: Alert fires $\rightarrow$ triage begins $\rightarrow$ failed intervention occurs $\rightarrow$ correct intervention succeeds. Traditional document chunking obliterates this sequence.
To fix this, we adopted the concepts outlined in what is agent memory. Our agent did not need a semantic search engine over raw logs; it needed an episodic memory bank containing causal relationships and negative feedback loops.
Code-Backed Implementation
Let's walk through how we implemented this in code.
1. Modeling Fix Attempts Relational Rigor
In our relational schema, we treat fix attempts as first-class entities with explicit outcome constraints. Here is the SQLAlchemy model definition from database/models.py:
class FixAttempt(Base):
__tablename__ = "fix_attempts"
id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True)
incident_id: Mapped[int] = mapped_column(
Integer, ForeignKey("incidents.id", ondelete="CASCADE"), nullable=False, index=True
)
action: Mapped[str] = mapped_column(Text, nullable=False)
outcome: Mapped[str] = mapped_column(String(32), nullable=False, index=True)
by_engineer: Mapped[Optional[int]] = mapped_column(
Integer, ForeignKey("engineers.id", ondelete="SET NULL"), nullable=True
)
at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
__table_args__ = (
CheckConstraint("outcome IN ('worked', 'failed', 'partial')", name="chk_fix_outcome"),
)
By constraining outcome directly in PostgreSQL, we guarantee that our operational dataset cleanly differentiates between remediations that succeeded and interventions that wasted critical minutes.
2. Distilling Incidents into Episodic Memories
Raw log lines remain in PostgreSQL. Under no circumstances do we stream raw log_lines to Hindsight. Instead, our extraction pipeline queries Postgres for the incident, its triggering alert, the most recent deployment preceding the outage, and all recorded fix attempts.
We then synthesize this into a structured memory block using the formatter in data/seed_hindsight.py:
def format_incident_memory(
incident: Dict[str, Any],
service_name: str,
alert_info: Optional[Dict[str, Any]],
recent_deploy: Optional[Dict[str, Any]],
fix_attempts: List[Dict[str, Any]],
) -> str:
code = incident["code"]
family = incident["root_cause_family"]
root_cause = incident["root_cause"]
duration = incident["duration_min"]
symptom = (
f"Alert fired for {alert_info['metric']} (value: {alert_info['value']}, "
f"threshold: {alert_info['threshold']})."
if alert_info else f"Service degradation in {service_name}."
)
deployment_desc = (
f"Deployment '{recent_deploy['summary']}' was pushed "
f"(config_change={recent_deploy['config_change']})."
if recent_deploy else "No immediate deployment preceded this incident."
)
failed_fixes = [f["action"] for f in fix_attempts if f["outcome"] == "failed"]
worked_fixes = [f["action"] for f in fix_attempts if f["outcome"] == "worked"]
return (
f"[{code}]\n[{service_name}]\n[family: {family}]\n\n"
f"Symptom:\n{symptom}\n\n"
f"Deployment:\n{deployment_desc}\n\n"
f"Failed fix:\n{'; '.join(failed_fixes) if failed_fixes else 'None'}\n"
f"Outcome:\n{'FAILED.' if failed_fixes else 'N/A'}\n\n"
f"Root cause:\n{root_cause}\n\n"
f"Successful fix:\n{'; '.join(worked_fixes)}\n"
f"Outcome:\nWORKED.\n\n"
f"Resolution time:\n{duration} minutes."
)
Notice the explicit presence of the Failed fix block. In operations, knowing what not to do is often more critical than knowing what to do. If an engineer knows that restarting pods failed and wasted 15 minutes in past outages of this type, they avoid repeating that mistake.
3. Ingestion and Temporal Retention in Hindsight
Once formatted, memories are retained into Hindsight using the official client SDK. We retain memories chronologically, attaching context tags and explicit document IDs to ensure idempotency:
from hindsight_client import Hindsight
client = Hindsight(base_url=base_url, api_key=api_key)
# Initialize memory bank with domain-specific mission
client.create_bank(
bank_id="kartly-sre",
name="Kartly SRE Operational Memory",
mission="Store and recall recurring incident failure patterns, root causes, "
"failed fixes, and successful remediations for Kartly microservices.",
)
# Retain memories preserving chronological timestamps and document identity
for item in memories_to_retain:
client.retain(
bank_id="kartly-sre",
document_id=item["document_id"],
content=item["content"],
timestamp=item["timestamp"],
context=item["context"],
tags=item["tags"],
metadata=item["metadata"],
)
According to the Hindsight documentation, providing explicit timestamp parameters allows the memory engine to build a temporal graph of facts, while document_id ensures that re-running pipelines updates existing episodes rather than cluttering the bank with duplicates.
4. Preventing Data Contamination with Dataset Partitioning
To benchmark our agent honestly, we divided our 110 operational incidents into two strictly isolated sets:
- Historical Memory Bank (INC-0001 to INC-0070): 70 incidents loaded into PostgreSQL and ingested into Hindsight. This represents the agent's accumulated operational experience over six months.
-
Held-Out Replay Dataset (INC-0071 to INC-0110): 40 incidents stored in PostgreSQL and exported to
data/replay/held_out/incidents.json. These incidents are strictly excluded from Hindsight.
When an incident from the replay dataset occurs, the agent must diagnose it solely by recalling analogous historical incidents that share underlying root-cause families (such as db_pool, redis_eviction, or tls_expiry).
Results and Concrete Interactions
To see the system in action, consider what happens when an alert fires for our primary checkout service.
Scenario: Flash Sale Connection Exhaustion
During a peak festival sale, an alert fires:
ALERT: db_connection_pool_active_ratio
Service: checkout-api
Value: 0.985 (Threshold: 0.90)
Payload: {"pool_name": "HikariCP-MainPool", "active_connections": 50, "waiting_threads": 42}
Without Episodic Memory (Baseline LLM / Naive Search):
The baseline assistant reviews recent logs showing JDBC timeout exceptions and suggests standard runbook steps:
- "Restart checkout-api deployment pods to clear stuck threads."
- "Flush Redis cache keys to reduce load."
The on-call engineer restarts the pods. While pods terminate, active connections drop to zero. But as soon as eight new pods boot simultaneously, each opens its minimum pool connections at the exact same instant, creating a thundering herd on PostgreSQL. The database connection slots saturate completely (FATAL: sorry, too many clients already), and checkout goes down entirely. Triage time: 48 minutes.
With Hindsight Episodic Memory:
The agent queries Hindsight using the alert symptom and service context. Hindsight recalls historical incident INC-0014, which occurred under identical load patterns:
RECALLED MEMORY: [INC-0014] [checkout-api] [family: db_pool]
Symptom: Alert fired for db_connection_pool_active_ratio (value: 0.99, threshold: 0.90).
Failed fix: Restarted service deployment pods (connections exhausted again within 2 minutes). Outcome: FAILED.
Root cause: HikariCP connection pool exhausted due to unclosed database connections during peak traffic surge.
Successful fix: Increased RDS Postgres max_connections to 400 and tuned HikariCP maximumPoolSize to 65. Outcome: WORKED.
Resolution time: 35 minutes.
The agent synthesizes this memory and outputs an evidence-backed recommendation:
Hypothesis (Rank 1): HikariCP connection pool exhaustion triggered by unclosed database connections under surge traffic.
Warning: Do NOT restart checkout-api pods. In INC-0014, restarting pods caused immediate re-exhaustion and prolonged downtime.
Recommended Action:
- Terminate idle backend PostgreSQL connections using
pg_terminate_backend().- Increase HikariCP
maximumPoolSizefrom 50 to 65.- Verify connection leak detection logs in
checkout_dao.py.
The engineer executes the targeted fix immediately. Active connection ratio normalizes within 4 minutes. Total triage time: under 8 minutes.
Lessons Learned
Building and operating this system taught us several concrete lessons about AI-assisted infrastructure operations:
1. Raw Logs Are Poison for Agent Memory
Dumping raw logs into an agent's memory bank degrades recall quality. Logs contain ephemeral noise (pod IPs, request IDs, memory addresses) that dilute semantic meaning. Extracting the structured causal arc—symptom, prior change, failed action, root cause, working fix—into clean markdown units improved retrieval accuracy across all eight failure families.
2. Negative Memory Is More Valuable Than Positive Memory
In production outages, engineers often jump to intuitive remedies: restart the service, clear the cache, or scale horizontal replicas. In distributed systems, these intuitive actions frequently trigger secondary failures (cache stampedes, connection storms, or replication lag). Recording failed fixes with explicit FAILED outcomes prevents agents from recommending disastrous shortcuts.
3. Relational Rigor and Semantic Memory Complement Each Other
We did not replace our relational database with an AI memory engine, nor did we try to force semantic reasoning into PostgreSQL. Postgres is unmatched at maintaining foreign key referential integrity, enforcing CHECK constraints, tracking ACID transactions, and storing high-volume log streams. Hindsight is unmatched at synthesizing historical human experience, managing temporal context, and retrieving relevant prior lessons. Using both in tandem gave us the best of both worlds.
4. Idempotency Must Be Baked Into Seeding Pipelines
In any real-world data pipeline, ingestion jobs will be triggered multiple times. Without deterministic document identifiers (document_id=f"kartly-{inc['code'].lower()}"), re-running ingestion creates duplicate memory entities that fragment agent attention. Designing idempotency into the schema and the memory API from day one saved dozens of debugging hours.
Looking Ahead
Engineering incident response is not a lack-of-data problem; it is a memory problem. The logs, alerts, and postmortems already exist in every modern engineering organization. By pairing a robust relational backbone with long-term episodic memory via Hindsight, we can ensure that when systems fail, our tools remember the lessons we already paid to learn.
Top comments (0)