How Hindsight Taught My DevOps Agent to Remember Outage

I started this project unable to run Python. pip wasn't recognized, PATH was broken, and I had inherited an async codebase I hadn't written. Hours later I refreshed a browser tab and watched a memory graph fill with real incident knowledge for the first time. This is what I built, what broke, and what I'd copy again.

The problem: incident knowledge walks out the door

Commonly cited figures say roughly 78% of production incidents are repeats of earlier failures, and that downtime costs $5,600 to $9,000 per minute for mid-size firms. I can't verify those numbers myself, but the pattern is familiar to anyone who has carried a pager. Somebody fixed this before. The fix lives in a Slack thread, or in the head of someone who has since left.

Alerts tell you what broke. Runbooks go stale. Postmortems get filed and forgotten. None of them remember.

What the platform does

My teammate Nishikanth and I built the foundation together: a modular monolith in Python/FastAPI with three specialist agents (incident response, code review, pipeline risk), an event queue, and a REST API. That base is solid, and everything below builds on it. I added the Hindsight memory loop and the Mission Control UI. Six of us worked on this as a team: Nishikanth and I on the build, Munishwar and Sankeerth on testing, and Prameela and Nikhil on the demo video.

The event flow:

webhooks (GitHub / GitLab / Jenkins / Datadog / PagerDuty / Jira)
   -> verify signature -> normalise -> enqueue (priority lanes)
   -> EventProcessors -> AgentOrchestrator -> specialist agent
   -> MemoryService (relational + pgvector + Hindsight)
   -> AgentSuggestion -> human feedback -> back into memory

Every agent follows the same three-step contract in app/agents/base.py: retrieve_context, generate_response, record_outcome. Because the orchestrator, the API, and the queue workers all call the same process(), adding memory to one place improved all three agents.

The stack: FastAPI, SQLAlchemy, PostgreSQL with pgvector, Redis, an OpenAI-compatible gateway, the Hindsight SDK, and a vanilla JS + Tailwind frontend.

Memory: hybrid, not just vectors

MemoryService pairs semantic recall with structured lookup:

contexts = await memory.find_similar_incidents(
    "database connection pool exhausted", service="payments", top_k=5
)
# each RetrievedContext: similarity + full Incident + its Runbook

The platform already had two local layers: a relational store for incidents, runbooks, and suggestions, and a pgvector table with cosine distance and an HNSW index. There was even a small feedback loop. Resolving an incident deletes the symptom-only embedding and re-embeds the resolved incident, so later alerts match on the fix.

That's useful, but it's per-database and flat. What I wanted from Hindsight agent memory was cross-session memory that consolidates over time. In the Hindsight console the bank's edges are typed: semantic, temporal, entity, and causal. Incidents are chains ("deploy, then config drift, then pool saturation, then page"), and a flat index can't represent that. You can read how it works in the Hindsight docs and the open-source repository.

Where retain happens

Nothing is stored automatically. The agent stores each suggestion together with the historical matches it used and waits for a human rating. Feedback arrives at POST /incidents/{id}/feedback, and that's where the outcome is composed into a memory string and retained to Hindsight:

# sketch of the idea; the real call lives in app/memory/service.py
memory_text = (
    f"Incident: {incident.title} ({incident.service})\n"
    f"Resolution: {resolution}\n"
    f"Rating: {rating}"
)
result = await hindsight.retain(bank_id="devops-ai", content=memory_text)
log.info("hindsight retain result: %s", result)

I retain outcomes, not chat transcripts. A transcript is mostly noise, and noisy memory produces noisy recall.

The bug that nearly beat me

Retain appeared to succeed. The cloud console stayed empty. I suspected the credentials, then the SDK, then my environment. In the meantime PowerShell kept mangling my environment variables, and CORS blocked the frontend from reaching the API.

What finally worked was a standalone ping script that used my credentials outside the app. It succeeded, which proved the key and the cloud were fine and the bug was mine. Tracing the execution path by hand showed that lazy initialization left the client unset, so the retain call was skipped silently. I fixed the wiring, added explicit log branches for "client missing," "retain sent," and "retain result," and refreshed the tab. The graph lit up.

If I had to compress the lesson into one line: an HTTP 200 is not proof of storage. Check at the destination.

The loop in action

Mission Control is a dark dashboard with a Hindsight memory graph, KPI cards, a suggestion-rating panel, and an execution log. The Simulate DB-Pool Failure button runs the whole narrative:

1. Create the incident. A payment-api connection-pool failure comes in as SEV1. The agent starts triaging and, on the first pass, recall pulls back prior symptom matches around 79%, drawn from the incidents already seeded in the bank.

1. Teach the fix. I rate the suggestion and submit the resolution. The toast confirms it: taught the agent, stored in Hindsight.

1. Watch the execution log. The demo runs through its steps in real time: human feedback and teaching, the Hindsight retain completing and indexing into the devops-ai bank, then a second, reworded outage on the same service triggering a fresh recall query.

1. Re-trigger a similar incident. The recalled matches carry the resolution forward, so the second incident doesn't start from zero.

Confidence in this system is derived, not invented: the average retrieval similarity plus a small prior, capped at 0.95. Low similarity triggers an explicit escalate action instead of a confident-sounding answer. So a ~79% match shows up as a strong hint with its source incident attached, not as a claim of certainty.

The README's offline mode uses a fake LLM provider, in-memory vectors, and an in-memory queue, so anyone can reproduce the loop without an OpenAI key, Redis, or pgvector.

What is real, and what isn't

This is a prototype. It hasn't been tested against production traffic. The 78 tests run offline against SQLite with a fake gateway and queue, and an end-to-end smoke script exercises health, auth, the incident lifecycle, agent queries, risk assessment, and a signed webhook. That verifies the logic. It doesn't measure real-world outcomes.

In particular, the dashboard's MTTR tile is a demonstration value. I haven't measured mean time to resolution, and I won't claim a number I haven't measured. The claim I'm comfortable making is narrower: recall gives an on-call engineer a starting point, with the prior fix and its source attached, instead of a blank terminal.

Why I think this direction matters

• Turnover stops erasing knowledge. Memory lives outside the agents and outside any one engineer.
• The expertise curve flattens. A first-week engineer sees the same past fixes as a veteran.
• Documentation becomes a compounding asset. Every rated incident improves the next one, unlike a static runbook.

The next step is connecting real PagerDuty webhooks (the adapter already exists) and measuring MTTR with and without recall over 90 days.

Lessons

1. Prove the dependency in isolation first. A standalone script cut the search space in half.
2. Log the presence of your client, the request, and the response. Lazy initialization fails quietly.
3. Put a human between a suggestion and memory. Otherwise the agent memorizes its own mistakes.
4. Similarity is not correctness. Show the source incident next to every suggestion.
5. Trace execution paths yourself. Abstractions save time until they hide the bug.

Thanks to the Vectorize/Hindsight team for making agent memory accessible without an enterprise contract.

Contributors

• Meghanadh Baipothu — Hindsight memory integration, Mission Control UI, testing
• Nishikanth Bonthu — platform architecture
• Munishwar Ravva — testing
• Sankeerth — testing
• Prameela Duvvari — testing & demo video
• Nikhil Arja — demo video

Code: github.com/manoj-gif-b/devops-ai