How We Built an Autonomous SRE Incident War Room on Switch

Ronit Gupta — Sat, 31 Oct 2026 07:59:55 +0000

How We Created an Autonomous SRE Incident Command Center Using Switch By integrating specialized AI agents, team messaging, and human approval steps, we reduced production downtime from 45 minutes to just 3 minutes.

The Reality of Production Incidents

Every Site Reliability Engineer and developer is familiar with this nightmare scenario: It is 9:42 AM, and PagerDuty alerts you that a checkout microservice is generating cascading HTTP 500 errors. Revenue is dropping as customers abandon their carts. The incident response begins in a Slack or Mattermost channel with ten engineers jumping in. While one person manually sifts through thousands of Datadog log lines, another monitors Kubernetes pod restarts, and others debate which commit caused the issue in version v2.14.0. By the time the team identifies the problematic commit and decides on a database-safe rollback, 40 minutes have elapsed. For SaaS or e-commerce companies, such delays result in significant financial losses.

Recognizing this gap during the Flip the Switch Hackathon, we realized that Switch offers the missing piece: the ability to transform individual AI coding assistants into collaborative team members within chat channels. We developed Incident-Sentinel, an autonomous Site Reliability Engineering workflow where AI agents and on-call engineers collaboratively triage, diagnose, approve, and resolve production outages in under three minutes.

Core Philosophy: AI as Teammates, Not Chatbots

Most developers currently utilize AI tools like Claude Code, Codex, or Cursor in isolation. However, production incidents are inherently collaborative events requiring three key elements:

Shared Context: All team members must view diagnoses and evidence in real-time.
Specialization: Separate agents should handle distinct tasks rather than relying on a single, generic prompt.
Human Oversight: AI agents must never execute production rollbacks without explicit human authorization.

Switch simplifies this architecture through Switch Rooms and Agent Templates.

System Architecture

The Incident-Sentinel workflow operates as follows:

Detection: An outage or error spike is reported in the #incident-war-room chat channel.
Triage: The @sentinel-triager agent queries telemetry via FastMCP, identifying error rates and correlating them with recent commits.
Remediation Planning: The @sentinel-remediator inspects the code changes, formulates a rollback strategy, and presents a human approval card.
Human Approval: The on-call engineer reviews the plan and authorizes the action.
Execution: The @sentinel-operator executes the rollback, verifies system health, and generates a post-mortem report.

Specialized Agents

@sentinel-triager: This agent automates telemetry analysis. It detects service degradation, calculates error rates and latency spikes, correlates these metrics with recent deployment SHAs, and isolates specific stack traces (e.g., NullPointerExceptions).
@sentinel-remediator: Once the root cause is identified, this agent assesses the blast radius by checking database schema compatibility. It then creates a safe rollback plan and posts an interactive card in the chat, requesting explicit "APPROVE" confirmation from the on-call engineer.
@sentinel-operator: Upon receiving human authorization, this agent executes the rollback, runs synthetic health probes to verify zero errors, and automatically compiles a standardized post-mortem report.

Implementation via Switch Templates

Switch uses declarative YAML configurations to set up these multi-agent environments. The template defines three distinct agents:

Triager: Responsible for telemetry queries and root cause isolation.
Remediator: Focused on mitigation planning and enforcing human approval gates.
Operator: Tasked with executing rollbacks, verifying health, and generating reports.

When instantiated in the Switch Console, this template creates a dedicated room in Slack or Mattermost, linking each agent to its specific workflow while posting a kickoff alert.

Reproducibility

To demonstrate that this system is fully reproducible, we built an enterprise infrastructure simulator using FastAPI. Users can test the entire incident response loop without needing live cloud access by cloning the repository and running the automated test script. The test successfully simulates detecting a degraded service, correlating it with a specific commit, executing an authorized rollback, and verifying full system recovery.

Key Takeaways

Collaborative AI: AI agents are more effective in shared team chat environments than in isolated IDE tabs.
Specialization Works: Dividing responsibilities among specialized triager, remediator, and operator agents creates more reliable workflows than monolithic prompts.
Human Oversight is Essential: The combination of autonomous diagnosis, human approval, and autonomous execution represents the best practice for AI in production SRE.

For the complete open-source code and template, visit github.com/ronitgupta138/Incident-Sentinel.

Built by Ronit Gupta for the Flip the Switch Hackathon.

Why We Built a Developer OS with 45 Native Apps That Idles Under 350MB RAM published: true

Ronit Gupta — Thu, 01 Oct 2026 09:09:53 +0000

Modern developer workstations have a hidden tax: background bloat.

Between Electron-wrapped API clients, container dashboards, background telemetry, and heavyweight desktop environments, a machine  
can easily consume 4GB to 8GB of RAM before you even write a line of code.

We engineered Aero Linux to solve this from first principles.                                                                      

────────────────────────────────────

The Core Philosophy: "Zero Bloat, Full Utility"

Aero Linux is an ultra-lean Linux distribution engineered specifically for developers, systems programmers, and low-latency
workflows.

Instead of bundling heavy cross-platform runtimes for basic tasks, Aero provides 45 native, zero-terminal GTK3 developer           
applications running on top of a streamlined desktop and dynamic memory compression engine.

📊 The Numbers:                                                                                                                    

• Idle RAM: < 350 MB (with full desktop UI loaded)
• Native Developer Apps: 45 standalone GTK3 applications
• Test Suite: 130 / 130 passing automated unit tests
• Memory Subsystem: Dynamic zRAM with ZSTD compression                                                                             
• Packaging: 1-command installer + automated Debian (.deb) packages

────────────────────────────────────                                                                                               

What’s Inside the Native 45-App Suite?

Developers often drop to the terminal not because they want to, but because GUI developer tools are either paid, bloated, or
non-existent on Linux.

Aero packages lightweight native utilities covering the full daily engineering lifecycle:                                          

1. API & Backend: Native REST/WebSocket API studio, JWT debugger, regex validator, and cURL converter.
2. Databases & Storage: SQLite/Postgres/Redis visualizers, connection testers, and schema inspectors.                              
3. Containers & Virtualization: Zero-overhead Docker/Podman container manager and image inspector.
4. Git & Code Velocity: Git branch graph, conflict resolver, changelog generator, and commit formatter.
5. AI & Inference: 1-click local LLM runner (Ollama daemon integration) with memory throttling.                                    
6. Hardware & Systems Telemetry: Real-time zRAM monitor, process inspector, audio/display switchers, and CPU frequency governor.   

All 45 apps share a unified dark theme, standard GTK3 styling, and near-zero memory footprints (each app idles at 10–25MB RAM).    

────────────────────────────────────

The Engineering: How We Kept It Fast

1. In-Memory ZSTD Compression
compiling, running containers, or local model inference.

2. Zero Runtime Bloat                                                                                                              

No background Electron wrappers, no telemetry daemons, and no mandatory web views. Every tool is built with native Python 3,       
PyGObject (GTK3), and direct Linux system APIs (/proc, sysfs, systemd).                                                            

3. Rigorous Test Coverage                                                                                                   

Every system component, script, and application is validated with a root make test pipeline running 130 automated unit tests
before any release is cut.                                                                                                         

────────────────────────────────────                                                                                        

Try It Out

Aero Linux is completely open source (MIT licensed). You can install the complete 45-app developer suite on any                    
Debian/Ubuntu/Mint machine with one command:

  ─ bash                                                                                                       
  curl -fsSL https://raw.githubusercontent.com/ronitgupta138/aero-linux/main/install.sh | bash

• GitHub Repo: github.com/ronitgupta138/aero-linux                                                                                 
• Web Documentation: ronitgupta138.github.io/aero-linux                                       

────────────────────────────────────                

What does your current idle RAM look like, and what developer GUI tool do you wish existed as a lean native app? Drop your
thoughts below!

DEV Community: Ronit Gupta