Building TrueSRE: Autonomous Multi-Agent Incident Remediation with TrueForge, FastMCP, and Daytona

💥 The 3:00 AM On-Call Nightmare
If you’ve ever carried an on-call pager for a production Kubernetes cluster, you know the dread of a 3:00 AM alert:

Your phone buzzes aggressively. The payment gateway is crashing with CrashLoopBackOff. Users are encountering 500 internal errors, and the business is losing thousands of dollars per minute.

You sit down half-asleep and start the manual diagnostic grind:

Open Prometheus and craft PromQL queries to check if the service is down (up == 0).
Sift through thousands of Loki log lines searching for JVM or Go panic stack traces.
Run kubectl describe pod to extract container termination exit codes (ExitCode: 137 OOMKilled).
Switch to GitHub to review recent commits and unified diffs to see who deployed what.
Formulate a hypothesis, draft a rollback command, and pray it doesn't break production further.
By the time the issue is resolved, 45 minutes have passed.

According to Gartner, enterprise IT downtime costs an average of $5,600 per minute ($300,000+ per hour).

Why Standard AI Chatbots Fail in Production
Generic LLM wrappers cannot solve this. When given access to a terminal, naive AI agents frequently hallucinate destructive commands (rm -rf /, dropping databases, or deleting the wrong namespaces) with zero security boundaries.

We asked ourselves: Can we build an autonomous, multi-agent SRE swarm that diagnoses and remediates real infrastructure in under 25 seconds—while guaranteeing 100% safety?

That is why we built TrueSRE.

🌟 Introducing TrueSRE
TrueSRE is an enterprise-grade autonomous Site Reliability Engineering multi-agent platform. Built on the TrueForge Agent Harness, official Model Context Protocol (FastMCP), Daytona Isolated Sandboxing, and Cryptographic HMAC-SHA256 Human-In-The-Loop (HITL) Safety Gates, TrueSRE takes outages from detection to verified resolution in under 25 seconds.

   🚨 3:00 AM Outage Detected (SEV-1 OOMKilled)
                       │
                       ▼

🧠 TrueForge Swarm Dispatched (PromQL, Loki, K8s, Git VCS via FastMCP)
│
▼
🔬 Bayesian Competing Hypotheses (4 Ranked Possibilities — 92% Confidence)
│
▼
📦 Daytona Isolated Sandbox Dry-Run (ExitCode: 0 Syntax Pre-Validation)
│
▼
🔐 Cryptographic HMAC-SHA256 Human-In-The-Loop Approval Gate
│
▼
⚡ FastMCP Mutating Rollout Undo + Closed-Loop Telemetry Verification
│
▼
📝 Auto-Generated Blameless Post-Mortem & Vector RAG Incident Memory
🏛️ System Architecture: The 4-Layer Foundation
TrueSRE is engineered with strict separation of concerns across 4 distinct layers:

1. Presentation Layer (Cyber-SRE Mission Control)
   A high-performance React 18 + Vite + Tailwind CSS dashboard that streams real-time agent thoughts, tool dispatches, live topology maps, and interactive Daytona terminals over WebSockets (ws://localhost:8000/ws/stream).

2. TrueForge Agentic Runtime (FastAPI Backend)
   The core intelligence engine. TrueForge coordinates turn-based agent execution, dynamic tool selection, and state transitions across a deterministic 15-node state machine.

3. Model Context Protocol (FastMCP) & Daytona Sandbox Boundary
   Decouples AI reasoning from raw infrastructure execution. An official FastMCP JSON-RPC server exposes 18 enterprise tools, while ephemeral Daytona sandboxes test remediation runbooks before touching live workloads.

4. Infrastructure & Telemetry Stack
   Direct integration with real Kubernetes clusters, Prometheus (http://localhost:9090), Grafana Loki (http://localhost:3100), Jaeger Tracing, and local Git VCS repositories.

🤖 The 6-Agent Specialized Swarm
Rather than relying on a single monolithic prompt, TrueSRE delegates responsibilities across 6 specialized agent personas:

Agent Persona Role in TrueForge Harness Key Responsibility
🛡️ Incident Commander Triage & State Orchestration Classifies severity (SEV-1 to SEV-4), coordinates agent lifecycles, and broadcasts state.
📈 Telemetry Investigator Metrics & Log Diagnostics Dispatches PromQL queries (up, rate, P95 latency) and queries Loki LogQL streams.
🔍 RCA Reasoner 5-Whys Root Cause & Git Blame Analyzes unified Git diffs and evaluates 4 competing hypotheses with probabilistic confidence.
🛠️ Remediation Planner Runbook Synthesis & Daytona Dry-run Formulates rollback runbooks and validates YAML schemas inside an isolated Daytona sandbox.
🔒 Safety HITL Gatekeeper Blast Radius & Cryptographic Gate Calculates downstream blast radius and enforces single-use HMAC-SHA256 operator authorization.
📝 Post-Mortem Scribe Blameless RCA & Incident Memory Auto-compiles markdown incident reports and stores vector embeddings in incident memory.
🛡️ Enterprise Safety: Daytona Sandboxing & Cryptographic HITL
Safety was our #1 architectural constraint. We implemented two bulletproof safety mechanisms:

1. Ephemeral Daytona Sandbox Dry-Runs Before executing a remediation command against Kubernetes, the Remediation Planner spins up an isolated Daytona sandbox container and dry-runs the command:

bash
kubectl rollout undo deployment/payment-service --dry-run=client -o yaml
If an agent or prompt injection attempts a destructive command like rm -rf /, TrueSRE's Security Policy Engine intercepts it instantly:

text
SECURITY_VIOLATION: Execution blocked by TrueSRE Daytona Policy.
Reason: Command matches catastrophic destruction pattern: rm -rf
Risk Level: FORBIDDEN | Exit Code: 126

1. Single-Use Cryptographic HMAC-SHA256 Authorization When an action modifies production, TrueSRE pauses at the Human-in-the-Loop (HITL) gate. Upon operator review and approval in the UI, the server issues a cryptographic token: hitl_

The FastMCP server validates this cryptographic token at the tool boundary. Unsigned requests are rejected with MUTATION_BLOCKED, and the token is revoked immediately upon execution.

🔄 End-to-End Walkthrough: Resolving a Real SEV-1 Outage in 20 Seconds
Let’s trace a real out-of-memory crash loop on our payment-service:

Detection: Prometheus fires an alert—payment-service pods are restarting with exit code 137 (OOMKilled).
Investigation: The Telemetry Investigator dispatches FastMCP tools to query Prometheus availability (up == 0) and extracts JVM heap overflow stack traces from Loki.
Root Cause Analysis: The RCA Reasoner inspects recent Git commits, finds commit #e89a10f in BufferPool.java, and discovers an unbounded memory cache regression.
Hypothesis Evaluation: The Bayesian reasoner scores 4 competing hypotheses, ranking the memory leak code regression at 92% confidence.
Daytona Dry-Run: The Remediation Planner tests k8s_rollout_undo in Daytona and confirms Exit Code: 0.
Human Approval: The operator clicks [Approve & Execute Remediation] in the Mission Control modal.
Closed-Loop Verification: FastMCP executes the rollout undo. TrueSRE re-queries Prometheus and mathematically proves:
Error Rate: $84.5% \rightarrow 0.04%$ ($\Delta -84.46%$)
P95 Latency: $1,840\text{ms} \rightarrow 92\text{ms}$ (95% drop)
Cluster Health: Restored to nominal (up == 1).
Blameless Post-Mortem: The Post-Mortem Scribe synthesizes a complete Markdown post-mortem report and stores vector embeddings in our Incident Memory store.
Total MTTR: Under 20 seconds.

💡 What We Learned & Key Takeaways
Decoupling Reasoning from Execution with MCP: The Model Context Protocol (MCP) proved to be an absolute game-changer. By treating infrastructure tools as standardized JSON-RPC endpoints, we could swap diagnostic tools without touching agent logic.
TrueForge Agent Harness Reliability: TrueForge’s turn-based orchestration allowed us to manage complex multi-agent handoffs between Telemetry, RCA, and Remediation planners seamlessly.
Safety Cannot Be an Afterthought: Autonomous agents must be bounded by sandboxed dry-runs and cryptographic authorization tokens. TrueSRE proves that autonomy and enterprise safety can coexist.
🔗 Try TrueSRE Yourself!
GitHub Repository: https://github.com/VishalRRajput/TrueSRE
Tech Stack: Python 3.11, FastAPI, TrueForge, FastMCP, Daytona, React 18, Vite, Kubernetes, Prometheus, Loki.
Built with ❤️ by Vishal Rajput for the TrueForge Agent Harness Hackathon.