ZeroDown: Building an AI-Powered On-Call Assistant & ChatOps Engine with Switch & MCP

The 3:00 AM Nightmare Every Developer Knows

Your phone buzzes frantically. Multiple Discord channels light up with duplicate messages: "Is the site down?", "Auth is failing!", "Can anyone checkout?".

Before you’ve even rubbed the sleep from your eyes, you're faced with:

Sifting through disconnected logs across cloud dashboards.
Trying to remember who is authorized to restart the gateway.
Running risky manual SSH commands while half-asleep.
Answering anxious teammates across multiple chat platforms.

We built ZeroDown to turn this chaos into a calm, automated, and secure workflow.

What is ZeroDown?

ZeroDown is a full-stack, autonomous on-call incident response and ChatOps engine. It pairs a high-frequency microservice monitoring pipeline with local AI agents via Switch Console and the Model Context Protocol (MCP).

Whenever downtime occurs, ZeroDown:

Detects the outage before your users notice.
Deduplicates teammate alerts across Discord and Telegram into a single active ticket.
Runs AI-driven root cause analysis directly against live HTTP failure traces and subsystem probes.
Enables In-Chat Safe Remediation: Engineers can review diagnostic findings and authorize recovery actions directly inside Discord with a simple command: @on-call-ai oncall approve.
┌────────────────────────┐
│ Monitored Application │
│ (e.g., CampusSpace) │
└───────────┬────────────┘
│ Health Checks (Every 30s)
▼
┌────────────────────────────────────────────────────────┐
│ ZeroDown Platform │
│ │
│ ┌──────────────────┐ ┌───────────────────┐ │
│ │ Monitor Engine ├──────────►│ API Gateway │ │
│ └──────────────────┘ └─────────┬─────────┘ │
│ │ │
│ ┌──────────────────┐ │ │
│ │ AI Triage Engine │◄────────────────────┤ │
│ └──────────────────┘ │ │
│ ▼ │
│ ┌──────────────────────────────────────────────────┐ │
│ │ Real-Time Next.js Dashboard │ │
│ └──────────────────────────────────────────────────┘ │
└───────────────────────────┬────────────────────────────┘
│
Model Context Protocol (MCP)
│
▼
┌───────────────────────────┐
│ Switch Console Core │
└─────────────┬─────────────┘
│ Bridged Channels
▼
┌────────────────────────┐
│ Discord & Telegram │
│ (@on-call-ai Bot) │
└────────────────────────┘
Architectural Deep Dive

ZeroDown was engineered from the ground up as a microservices architecture designed for resilience:

1. Real-Time Monitor Engine (services/monitor)

A non-blocking probe worker that pings target application health endpoints every 30 seconds. It tracks latency, HTTP status codes, and individual subsystem health checks (e.g. database, auth, redis). When consecutive failures exceed the configured threshold, it automatically opens an incident.

1. The AI Triage & Diagnostic Engine (services/triage)

When an outage is declared, the triage service collects:

Recent probe failure traces (e.g., HTTP 503, connection timeouts).
Subsystem health checks.
Unverified teammate chat reports from Discord and Telegram.

It structures these inputs and runs root-cause analysis, returning:

Confirmed Facts vs. Hypotheses
Confidence Score
Recommended Action
Strictly Typed Recovery Proposal (manual_operator_action)

1. Bridging Local AI to Discord with Switch Console & MCP

Traditional chatbots require hosting webhook servers, managing Discord bot websockets, and writing custom gateway adapters.

Instead, we used Switch Console to bridge an OpenCode agent (on-call-ai) directly to Discord and Telegram. We built a custom Model Context Protocol (MCP) server (oncall-mcp.js) running over stdio that exposes safe incident investigation and ChatOps tools to the LLM:

oncall_get_incident: Queries current incident status without requiring UUIDs.
oncall_investigate_incident: Triggers root-cause triage on demand.
oncall_approve_incident_recovery: Validates operator permissions and records signed remediation approvals.
oncall_format_incident_reply: Converts raw JSON diagnostics into Discord-friendly Markdown summaries.
How It Works in Practice: A Real Outage Walkthrough

Here is what an incident looks like in action:

Step 1: Automated Detection

The production database connection drops. The monitor detects three consecutive 503 responses and triggers an incident ticket on the Next.js Dashboard.

Step 2: Teammate Interaction in Discord

An engineer sees an alert in the team’s #general channel and checks in:

Rishi [MLH]: @on-call-ai oncall status

on-call-ai [APP]:
🔴 Active Incident: 3e2b1a66-e329-46b9-8f2d-5fb0b316e2f0
• Application: Demo Application / Campus Card
• Severity: High · Status: Open
• Detected: 3 consecutive probe failures (HTTP 503)
• Investigation: Ready to investigate. Reply with @on-call-ai oncall investigate.

Step 3: In-Chat AI Root Cause Analysis

The engineer requests an investigation:

Rishi [MLH]: @on-call-ai oncall investigate

on-call-ai [APP]:
🔍 Investigation Results:
• Summary: Demo Application is degraded due to database check failures.
• Probable Cause: Internal failure in subsystem: database (unconfirmed).
• Confidence: 75%
• Proposed Action: Restart service process to clear transient connection pool exhaustion.
• To Authorize: Reply with @on-call-ai oncall approve.

Step 4: Direct ChatOps Approval

Rather than logging into cloud consoles or running shell scripts, authorized operators approve the remediation right in the thread:

Rishi [MLH]: @on-call-ai oncall approve

on-call-ai [APP]:
✅ Recovery Proposal Approved
• Incident: 3e2b1a66
• Authorized Action: manual_operator_action (Restart service process)
• Authorized By: Switch On-Call Bot (rishi09509)
• Status: Pending execution by remediation runner.

Key Challenges & Lessons Learned
Clean Failure Fallbacks: During local development and hackathons, remote databases or AI services can occasionally be unreachable. We implemented smart in-memory fallback repositories and deterministic heuristic root-cause inference engines. If OpenCode or Docker is momentarily offline, ZeroDown continues running smoothly without dropping incidents.
UUID-Free Frictionless ChatOps: Nobody wants to type a 36-character UUID on a mobile phone during a 3:00 AM incident. We built smart active-incident resolution so plain commands like oncall status, oncall investigate, and oncall approve automatically infer the ongoing ticket.
Safety First: AI agents should never execute destructive production commands autonomously. ZeroDown enforces human-in-the-loop approvals with cryptographically signed tokens and role-based permissions before any recovery action is authorized.
What’s Next for ZeroDown?
Dynamic Multi-Website Management: Adding a one-click dashboard modal to register and manage multiple URLs dynamically.
Proactive Outbound Alerting: Expanding Switch’s event bus to push unsolicited proactive alerts into bridged channels the second an anomaly is detected.
Automated Canary Rollbacks: Connecting recovery approvals directly to Kubernetes and GitHub Actions deployment runners.
Conclusion

Incident response shouldn't require juggling ten tabs, copying logs, and panicking over terminal windows. By pairing real-time observability with local AI agents via Switch and MCP, ZeroDown brings SRE-level automation directly to where teams already communicate.

Built with Next.js, Node.js, OpenCode, Switch Console, and Model Context Protocol (MCP).