How We Built an Autonomous SRE Incident War Room on Switch

How We Created an Autonomous SRE Incident Command Center Using Switch By integrating specialized AI agents, team messaging, and human approval steps, we reduced production downtime from 45 minutes to just 3 minutes.

The Reality of Production Incidents

Every Site Reliability Engineer and developer is familiar with this nightmare scenario: It is 9:42 AM, and PagerDuty alerts you that a checkout microservice is generating cascading HTTP 500 errors. Revenue is dropping as customers abandon their carts. The incident response begins in a Slack or Mattermost channel with ten engineers jumping in. While one person manually sifts through thousands of Datadog log lines, another monitors Kubernetes pod restarts, and others debate which commit caused the issue in version v2.14.0. By the time the team identifies the problematic commit and decides on a database-safe rollback, 40 minutes have elapsed. For SaaS or e-commerce companies, such delays result in significant financial losses.

Recognizing this gap during the Flip the Switch Hackathon, we realized that Switch offers the missing piece: the ability to transform individual AI coding assistants into collaborative team members within chat channels. We developed Incident-Sentinel, an autonomous Site Reliability Engineering workflow where AI agents and on-call engineers collaboratively triage, diagnose, approve, and resolve production outages in under three minutes.

Core Philosophy: AI as Teammates, Not Chatbots

Most developers currently utilize AI tools like Claude Code, Codex, or Cursor in isolation. However, production incidents are inherently collaborative events requiring three key elements:

1. Shared Context: All team members must view diagnoses and evidence in real-time.
2. Specialization: Separate agents should handle distinct tasks rather than relying on a single, generic prompt.
3. Human Oversight: AI agents must never execute production rollbacks without explicit human authorization.

Switch simplifies this architecture through Switch Rooms and Agent Templates.

System Architecture

The Incident-Sentinel workflow operates as follows:

1. Detection: An outage or error spike is reported in the #incident-war-room chat channel.
2. Triage: The @sentinel-triager agent queries telemetry via FastMCP, identifying error rates and correlating them with recent commits.
3. Remediation Planning: The @sentinel-remediator inspects the code changes, formulates a rollback strategy, and presents a human approval card.
4. Human Approval: The on-call engineer reviews the plan and authorizes the action.
5. Execution: The @sentinel-operator executes the rollback, verifies system health, and generates a post-mortem report.

Specialized Agents

1. @sentinel-triager: This agent automates telemetry analysis. It detects service degradation, calculates error rates and latency spikes, correlates these metrics with recent deployment SHAs, and isolates specific stack traces (e.g., NullPointerExceptions).
2. @sentinel-remediator: Once the root cause is identified, this agent assesses the blast radius by checking database schema compatibility. It then creates a safe rollback plan and posts an interactive card in the chat, requesting explicit "APPROVE" confirmation from the on-call engineer.
3. @sentinel-operator: Upon receiving human authorization, this agent executes the rollback, runs synthetic health probes to verify zero errors, and automatically compiles a standardized post-mortem report.

Implementation via Switch Templates

Switch uses declarative YAML configurations to set up these multi-agent environments. The template defines three distinct agents:

• Triager: Responsible for telemetry queries and root cause isolation.
• Remediator: Focused on mitigation planning and enforcing human approval gates.
• Operator: Tasked with executing rollbacks, verifying health, and generating reports.

When instantiated in the Switch Console, this template creates a dedicated room in Slack or Mattermost, linking each agent to its specific workflow while posting a kickoff alert.

Reproducibility

To demonstrate that this system is fully reproducible, we built an enterprise infrastructure simulator using FastAPI. Users can test the entire incident response loop without needing live cloud access by cloning the repository and running the automated test script. The test successfully simulates detecting a degraded service, correlating it with a specific commit, executing an authorized rollback, and verifying full system recovery.

Key Takeaways

1. Collaborative AI: AI agents are more effective in shared team chat environments than in isolated IDE tabs.
2. Specialization Works: Dividing responsibilities among specialized triager, remediator, and operator agents creates more reliable workflows than monolithic prompts.
3. Human Oversight is Essential: The combination of autonomous diagnosis, human approval, and autonomous execution represents the best practice for AI in production SRE.

For the complete open-source code and template, visit github.com/ronitgupta138/Incident-Sentinel.

Built by Ronit Gupta for the Flip the Switch Hackathon.