CI Watchdog — The Autonomous CI Minutes & Spend Sentinel with Open AI Failure Triage

Tushar Khadde — Mon, 05 Oct 2026 03:38:55 +0000

CI Watchdog

An autonomous GitHub App and SaaS platform that stops wasted CI spend before it hits your invoice, and automatically diagnoses broken builds using open-weight AI.

What I Built

Every developer has that one friend or team lead who dreads the 20th of the month: the day GitHub sends the notification:

"You have used 100% of your included GitHub Actions minutes."

I built CI Watchdog for my friend and engineering teammate, who was constantly battling ballooning CI bills and jammed pipeline queues on open-source and startup projects.

The problems:

Developers push 4 quick commits to a PR, and GitHub blindly spawns 4 redundant 15-minute test suites in parallel on expensive runners (Ubuntu, macOS, Windows).
When a build fails after 20 minutes, developers have to dig through 30,000 lines of terminal logs just to discover a blocked database container port or a missing env variable.

The solution:

Autonomous Waste Prevention: Detects and cancels superseded runs, runaway jobs exceeding historical p95 execution baselines, and PR workflows left running on closed branches.
Proactive Workflow Hygiene: Audits repository workflows for missing concurrency blocks and generates 1-click YAML fixes.
AI Failure Investigation: The moment a build fails, an integrated AI agent pulls the raw runner logs, redacts secrets, diagnoses the exact root cause, pinpoints the failed step, categorizes the bug, and provides copy-paste solutions.
Human-in-the-Loop Control Plane: Includes a kill switch, dry-run observation safety rails, team approvals for cancellations, and interactive ROI spend simulators.

Demo

🌐 Live Dashboard: ci-watchdog.vercel.app
🧪 Live Demo Pull Request: Live Demo

Key Features to Explore

Route	Feature	Description
`/app`	Control Room Overview	Live and dry-run minutes-saved counters, real-time system status indicators, and weekly savings charts.
`/app/investigations`	AI Investigations	Instant root-cause diagnostic cards generated by Gemma / Gemini 3.8 Flash, with failure hypotheses, log evidence snippets, and recommended next steps.
`/app/decisions`	Decision Engine & CSV Export	Full searchable decision history with rule and status filters, plus 1-click CSV download for engineering management reports.
`/app/approvals`	Human Approval Queue	First-N quota approval flow with 1-click Approve and Deny buttons.
`/app/digest`	Interactive ROI Calculator	Real-time savings simulator modeling monthly minute volume and runner tiers (see pricing below).

Runner pricing used in the ROI calculator:

Runner	Cost per minute
Ubuntu	$0.008
Windows	$0.016
macOS	$0.080
Apple Silicon (M-series)	$0.160

Code

The entire codebase is open-source and structured as a high-performance TypeScript monorepo using Turborepo and pnpm:

👉 Tusharkhadde/CI-watchdog

Path	Description
`apps/watchdog`	Event-driven Node.js background service deployed on Render. Connects to the GitHub App webhook stream, processes workflow states, runs the pure rule engine, and coordinates AI triage.
`apps/dashboard`	Next.js 16 (Turbopack) dashboard deployed on Vercel with Shadcn UI, Auth.js (OAuth with GitHub & Google), and real-time state synchronization.
`config/watchdog.yml`	Declarative team policies with per-repository overrides.

How I Built It

Architecture Overview

1. Pure, Deterministic Rule Engine

The rule engine (apps/watchdog/src/rules/) was designed to be 100% pure with zero side effects. Given the current run, sibling runs, historical durations, and workflow YAML, it evaluates four rules:

Rule	Behavior
`SUPERSEDED`	Cancels older runs queued on the same ref when a newer commit arrives.
`RUNAWAY_DURATION`	Compares running jobs against historical p95 execution times and alerts/cancels if a job hangs.
`MISSING_CONCURRENCY`	Scans workflow ASTs and alerts if jobs lack top-level concurrency cancellation blocks.
`STALE_ON_CLOSE`	Immediately terminates active pipelines when a PR is merged or closed.

2. Intelligent AI Failure Triage

When a workflow concludes with a failure, CI Watchdog activates the AI investigator:

Fetches the raw runner logs via the GitHub API.
Runs pattern redaction to scrub tokens and sensitive credentials.
Dispatches the condensed failure excerpt to open-weight models (Google Gemma / Gemini 3.8 Flash via OpenAI-compatible endpoints).
Validates the output with Zod, enforcing structured JSON containing:

Field	Description
`rootCause`	Explicit technical diagnosis (e.g., missing Docker service container, database port refusal).
`failedStep`	Exact bash command or step name that exited non-zero.
`category`	One of: Infrastructure, Configuration, Test, or Code.
`nextSteps`	Concrete remediation actions.

3. Safety Rails & Human-in-the-Loop

Automated cancellation can be intimidating for developers, so I implemented strict safety rails:

Default Dry-Run Mode: Repositories begin in observation mode so teams can review what would have been canceled without impacting active builds.
48-Hour Observation Window: Go-live can only be unlocked after a repository has been observed.
First-N Live Approval Quota: The first cancellations require explicit human confirmation via dashboard or Slack buttons.
Protected Branch Immunity: Never cancels main, master, or release/* branches.
Global Kill Switch: Instantly pauses all active cancellations across all repositories.

Why Does Open Innovation Matter?

Open innovation is what makes a tool like CI Watchdog possible without compromising developer trust:

Data Sovereignty & Privacy: Build logs often contain internal paths, dependencies, proprietary package names, and sensitive environment details. Relying on closed, proprietary cloud AI black boxes creates data-leakage risks for enterprise codebases. Open-weight models like Gemma enable teams to run AI failure analysis locally or within their own private infrastructure.
Deterministic Customization: Open tooling and open protocols allow developers to customize rule thresholds, inspect prompts, tune temperature, and swap backend models based on compute constraints.
No Vendor Lock-In: Developers shouldn't need a six-figure enterprise contract just to know why their CI pipeline broke or to prevent redundant job execution. Open innovation democratizes cost control for individual creators, open-source maintainers, and startups alike.

Prize Categories

Build for a Friend
Open Innovation & Open-Weight AI

Built with ❤️ during Hacktoberfest. If your team is burning minutes on redundant CI runs, give CI Watchdog a spin!

DEV Community: Tushar Khadde