DEV Community

Naman Bhatt
Naman Bhatt

Posted on

Turning Solo AI Agents into an SRE Incident Squad with Switch

#ai #devops #automation #sre

Every engineering team knows the dread of the 2:00 AM PagerDuty alarm: checkout is throwing HTTP 500 errors, customers are abandoning carts, and the slack channel is filled with panicked questions.

AI coding agents like Claude Code, Codex, and OpenCode have revolutionized solo developer productivity. But when production breaks, solo agents struggle:

  1. They exist in an isolated terminal on one developer’s laptop.
  2. They hallucinate causes from coincidental log lines without empirical testing.
  3. They lack shared state with human teammates.
  4. Giving an unconstrained LLM write access to production infrastructure is an SRE nightmare.

We built IncidentPilot to solve this: an autonomous, human-supervised engineering incident response squad running directly inside team chat via Switch.

1. Why Switch is the Missing Piece for Agentic Teams

Switch is open-source infrastructure that brings AI agents into team chat (Mattermost, Slack, Discord, Microsoft Teams). What makes Switch fundamentally different from a simple chatbot webhook:

  • Shared Room Context: Humans and agents read from the exact same timeline of messages, attachments, and decisions.
  • Room Documents: Living state (incident war-boards, post-mortems) stored natively in the room via create_room_document.
  • Role Leases (assume_role): Jobs like @sre-commander or @sre-debugger are held as distributed leases, preventing split-brain race conditions.
  • Wire-Level Agent Protocol: Server-Sent Events (SSE) streaming down, HTTP operations up, and 2-second heartbeats ensuring true distributed resilience.

2. The Squad Architecture

Inside #sre-incident-war-room, four specialized agents collaborate:

  1. @sre-commander: Listens for alerts, assigns severity (P1–P3), pins a live incident document, and manages the state machine.
  2. @sre-diagnostician: Slices container logs, queries Prometheus metrics, and retrieves relevant runbooks via Hybrid BM25+Dense RRF search.
  3. @sre-debugger: Runs automated tests in an isolated microservice sandbox (orders-api), reproduces the exact failure trace, and drafts a surgical patch diff.
  4. @sre-critic: An adversarial reviewer (ported from incident-suite, MIT) that cross-examines the diagnosis to catch hallucinations and checks the patch for regression risks.

3. Engineering Invariants: Beyond "AI Vibe-Coding"

To make IncidentPilot production-ready, we enforced three core engineering invariants:

A. Empirical Causal Triangulation

A diagnosis is rejected unless it can be empirically falsified in the sandbox:

  • The regression test must fail against the broken service.
  • The proposed patch must turn the exact same test green.

B. The Cryptographic Four-Eyes Approval Gate

No automated agent can mutate production without human sign-off.
When a patch is proposed, IncidentPilot computes a SHA-256 action digest:
$$\text{Digest} = \text{SHA-256}(\text{ActionType} \parallel \text{Target} \parallel \text{PatchDiff} \parallel \text{Timestamp})$$
When an on-call engineer signs off (@sre-commander approve), the engine re-evaluates the live hash. If anything changed between proposal and sign-off, execution safely aborts (TOCTOU defense).

C. Compensating Saga Rollbacks

Every approved remediation registers an automated rollback action. If post-remediation health probes fail within 30 seconds, the system automatically rolls back the change.

4. How to Get Started with Switch in 10 Minutes

If you’re setting up Switch for the first time:

# 1. Download Switch Console
# Available for macOS, Windows, and Linux from github.com/sandbox-quantum/switch

# 2. Start the local server
docker compose -f switch/deploy/standalone-docker-compose.yml up -d

# 3. Create your incident room from template
switch room create --template switch/templates/incident-pilot.template.yaml
Enter fullscreen mode Exit fullscreen mode

5. Evaluation Benchmarks

We evaluated IncidentPilot across 5 ground-truth production failure scenarios:

  • bad-deploy: HTTP 500 regression in checkout
  • cache-down: Redis stop leading to DB pool exhaustion
  • kill-worker: Worker crashed, Kafka consumer lag
  • oom-worker: Memory leak trigger
  • poison-message: Malformed JSON event crashloop

Results:

  • MTTR Reduction: >93% compared to manual human triage baselines.
  • Safety Compliance: 100% (zero unapproved production mutations).

Solo agents helped us write code faster. Switch turns them into true engineering teammates.

Top comments (0)