WARDEN: Hunting the Ghost in the Machine with an AI-Powered Self-Healing DevOps Agent

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 4: Ghost in the Machine

What I Built

WARDEN is an AI-powered self-healing DevOps agent that detects service failures, investigates their root causes, proposes safe remediation actions, and verifies whether the system has actually recovered.

Every production system has its ghosts.

A sudden traffic spike. An exhausted connection pool. Unexpected HTTP 503 errors. A service that appears healthy but silently struggles under concurrent load.

These failures can be difficult to diagnose. Engineers often spend valuable time switching between logs, metrics, configuration files, and monitoring dashboards.

I wanted to explore a different approach:

What if an AI agent could investigate incidents and help resolve them while keeping safety and human approval at the center?

That idea became WARDEN.

WARDEN follows a four-stage incident response workflow:

Detect → Diagnose → Remediate → Verify

Key Features

• Incident Detection: Monitors service health, telemetry, logs, latency, and HTTP failure rates.
• AI-Powered Diagnosis: Analyzes incident information and identifies potential root causes.
• Safe Remediation: Restricts proposed actions to a predefined allowlist with strict parameter boundaries.
• Human-in-the-Loop Approval: Requires operator authorization before applying configuration changes.
• Empirical Recovery Verification: Uses concurrent HTTP requests to determine whether the service has genuinely recovered.
• Audit Trail: Maintains structured records of incident and remediation events.

The Ghost in the Machine

For the demonstration, WARDEN manages a FastAPI-based payment-service simulator.

I can deliberately introduce connection-pool exhaustion, causing the service to return HTTP 503 errors under concurrent traffic.

WARDEN then:

1. Detects the service degradation.
2. Analyzes metrics and logs to identify connection-pool starvation.
3. Proposes an increase to the connection-pool limit.
4. Waits for explicit operator approval.
5. Applies the permitted configuration change.
6. Runs another concurrent HTTP verification to measure recovery.

The important part is that WARDEN does not simply announce that an incident is fixed because a configuration changed.

It verifies recovery using actual HTTP responses.

Demo

Watch WARDEN in action:

▶️ Watch the WARDEN Project Demo on YouTube

The demonstration showcases the project's incident-response workflow and recovery capabilities.

Code

The project is open source and available on GitHub:

WARDEN - AI-Powered Self-Healing DevOps Agent

Technology Stack

• Python
• FastAPI
• Streamlit
• Nebius Token Factory
• OpenAI-compatible SDK
• Pydantic
• HTTPX
• Pytest
• Tavily (optional technical research integration)

Project Architecture

WARDEN/
│
├── warden/
│   ├── app.py
│   ├── payment_service.py
│   ├── agent.py
│   ├── nebius_client.py
│   ├── tavily_client.py
│   ├── verifier.py
│   ├── remediation.py
│   ├── schemas.py
│   └── audit.py
│
├── tests/
├── requirements.txt
├── pytest.ini
└── README.md

The repository includes automated tests for service behavior, remediation safety, agent output validation, and empirical recovery verification.

How I Built It

WARDEN combines an AI reasoning layer with deterministic execution controls.

1. AI Reasoning

The agent architecture supports open-weight models through Nebius Token Factory's OpenAI-compatible API.

The documented model configurations include:

• Meta-Llama-3.1-8B-Instruct for lower-latency diagnostic tasks.
• Meta-Llama-3.1-70B-Instruct for more complex incident analysis.
• DeepSeek-R1 as an alternative reasoning model configuration.

The architecture separates fast triage from deeper reasoning so that model selection can match the complexity of an incident.

A deterministic Demo Mode is also available, allowing the incident-response workflow to operate without API credentials.

2. Safety Before Automation

Giving an AI agent unrestricted access to production infrastructure would introduce serious operational risks.

WARDEN takes a constrained approach.

The agent cannot execute arbitrary shell commands, Python code, SQL, or unrestricted operations.

Instead, remediation is limited to explicitly permitted actions:

• increase_pool_limit
• reset_service_config
• no_op

Numeric boundaries are enforced through deterministic validation, and configuration changes require explicit operator approval.

3. Real Recovery Verification

WARDEN uses HTTPX and asynchronous requests to evaluate service behavior under concurrent load.

It measures actual HTTP responses and performance indicators instead of trusting the agent's claims.

A proposed fix is not the same thing as a verified recovery.

4. Testing

The project includes 18 automated tests covering:

• Service health and concurrency behavior.
• HTTP 503 responses under overload.
• Administrative authentication.
• Remediation allowlists and parameter boundaries.
• Human approval enforcement.
• Structured diagnostic validation.
• Recovery verification.

Why Does Open Innovation Matter?

Open innovation matters because infrastructure automation should not depend on a single closed model or an opaque decision-making system.

Using open-weight model options through an inference platform such as Nebius Token Factory provides flexibility in model selection while allowing developers to retain control over the surrounding application architecture.

For WARDEN, the reasoning model is only one component of the system.

The application also contains independently testable safety validators, structured schemas, deterministic remediation rules, and recovery verification.

This separation makes it possible to experiment with different models without giving them unrestricted control over infrastructure operations.

Open innovation also makes it easier for developers to inspect, adapt, test, and extend the system according to their own operational requirements.

My Agent Session

WARDEN's incident lifecycle can be demonstrated through its dashboard:

Incident → Diagnosis → Approval → Remediation → Verification

The Spirit of It

The most interesting part of building WARDEN was realizing that an AI agent should not be judged only by how intelligently it explains a problem.

It should also be judged by what happens after its recommendation.

Does the service actually recover?

Can the action be trusted?

Can an engineer understand what changed?

And can the system prove that the incident is resolved?

I wanted to build something that goes beyond an AI chatbot suggesting commands.

WARDEN represents an approach where AI reasoning works together with deterministic safety, human oversight, and measurable outcomes.

The ghost in the machine might be invisible, but its impact is measurable.

WARDEN: Find the failure. Understand the cause. Fix it safely. Prove the recovery.