I Gave My SRE Agent Hindsight. Its Commands Got Safer.

Manya Agarwal — Sat, 07 Nov 2026 06:38:28 +0000

At 3 AM, a generic AI assistant will tell you to “check your database connection.” That is not an answer to a Postgres deadlock on a checkout table. Worse, it will sometimes hand you a destructive command with full confidence and no idea what your environment looks like.

I built NerveCenter to fix that. It’s an on-call agent that answers production alerts using what my team has already learned from past outages, instead of what a language model remembers from the internet.

What NerveCenter does

NerveCenter is a Streamlit dashboard with a chat box. You type an alert like Redis OOM Error or Kubernetes pods failing with CrashLoopBackOff. The agent then:

Looks up similar past incidents in a memory layer built around Hindsight
Shows the retrieved incident record in the UI so you can verify it
Sends that record to an LLM (openai/gpt-oss-120b on Groq) with strict instructions to base its answer on it
Returns exact terminal commands plus the post-mortem warnings from last time

The stack is small on purpose: Streamlit for the UI, Groq for fast inference, and an agent memory layer for institutional knowledge.

The story: memory is the difference between a textbook and a teammate

The most important design decision was putting a switch in the UI. One toggle in the sidebar turns memory on or off:

python
use_hindsight = st.sidebar.toggle("🧠 Enable Hindsight Memory", value=True)

With memory off, the agent gets this system prompt and nothing else:

python
system_prompt = (
"You are an elite Site Reliability Engineer (SRE) AI. "
"Provide exact terminal commands, root cause analysis, and post-mortem advice for server outages. "
"Structure your response with clear markdown headings."
)

That produces competent but generic advice. It’s what any assistant would give.

With memory on, the agent first recalls a past incident and injects it into the prompt:

python
if retrieved_context:
system_prompt += (
f"\n\nCRITICAL HINDSIGHT MEMORY:\n"
f"You MUST base your solution strictly on this past incident memory. "
f"Quote the specific commands used and highlight the post-mortem warnings:\n"
f"{retrieved_context}"
)

Now the model isn’t guessing. It’s working from a record that looks like this:

[Hindsight ID: INC-892 | Severity: P1]
Symptom: Redis OOM Error on Primary API Cluster.
Root Cause: Stale session keys failed to expire due to a misconfigured TTL policy.
Resolution: Executed redis-cli FLUSHALL ASYNC. Scaled ElastiCache node to r6g.2xlarge.
Post-Mortem: The FLUSHALL command caused a 12-second latency spike.
Route traffic to standby cluster before flushing next time.
Why the commands get safer

This is the part I care about most. The fix for the Redis incident was FLUSHALL, which is dangerous on its own. A generic assistant might suggest it without a word of warning. But the stored incident includes the lesson from last time: it caused a 12-second latency spike, so route traffic to a standby cluster first.

Because the prompt tells the model to quote the post-mortem, that warning arrives together with the command.

The Kubernetes case works the same way. The memory says a CrashLoopBackOff came from a liveness probe timing out against a third-party payment gateway, fixed by raising the timeout from 3s to 10s. The post-mortem adds the real lesson: don’t fail liveness probes on external API latency, use readiness probes for external dependencies. That is the kind of knowledge that lives in one senior engineer’s head. Memory moves it into the tool.

Transparency was a deliberate choice

The retrieved record is shown in an expander before the answer:

python
with st.expander("🔍 View Retrieved Hindsight Memory Log", expanded=True):
st.code(retrieved_context.strip(), language="log")

During an outage, an engineer shouldn’t have to trust a black box. If the agent says “terminate PID 4821,” I want to see the incident it came from. Showing the source makes the answer checkable in seconds.

What I’d be honest about

I want to be direct about the current state, because I’d want that from an article like this:

The memory store is still seeded by hand. Today the recall step is a keyword match over three incident records. It works for a demo flow, but it doesn’t generalize to an alert phrased differently. Moving this to Hindsight’s retain and recall, where incidents are stored as they happen and retrieved by meaning, is the next step. The recall(query) interface was built so that swap touches one class.
Some dashboard numbers are placeholders. The “32ms ping” metric is static, and a short sleep simulates database latency. I’d rather say so than let a screenshot imply otherwise.
The model still needs guardrails. I set temperature=0.1 to keep it from improvising, but “you MUST base your solution on this memory” is a prompt instruction, not a guarantee. A real deployment should validate generated commands before anyone runs them.
Lessons
Memory is about warnings as much as answers. The most valuable field in each record is the post-mortem, because it prevents repeat mistakes.
Put a memory toggle in your UI. The before/after comparison sells the idea better than any explanation.
Show retrieved context to the user. It builds trust and makes bad retrievals easy to spot.
Design the memory interface first. Keeping recall behind one method means the storage backend can change without touching the rest of the agent.
Be upfront about what’s placeholder. Engineers forgive limitations. They don’t forgive surprises.

If you’re building agents that need to learn from past events, start with the Hindsight docs and the GitHub repo. The code for NerveCenter is at github.com/nagurukethankumarreddy-alt/nervecenter-sre-agent.

DEV Community: Manya Agarwal

I Gave My SRE Agent Hindsight. Its Commands Got Safer.