Self-Healing IT: How AI Agents Could Have Prevented the California DMV Outage

Quick read · 7 min read

This article shows how to design self-healing systems that fix problems fast without creating new ones.

Key takeaways

1. Self-healing means finding the problem, deciding on a fix, applying it, and checking the result, not just restarting things.
2. Risky fixes need a person to approve them before the system makes any change.
3. Test your healing system with fake failures before you let it touch real systems.
4. Measure how often the healing system causes new problems, not just how fast it fixes old ones. <!-- omnithium-quick-read:end -->

The operating problem

Self-healing agents cut mean time to recovery only when they run inside bounded autonomy, observable decision paths, and reversible actions. A DMV-style network outage makes that constraint concrete. The California DMV outage that spiked search interest in September 2026 is a useful stress test, even if we don't know the exact root cause. We're not claiming any specific failure. We're asking a harder question: what would a well-designed self-healing system have done differently?

Most teams confuse self-healing with auto-remediation. Auto-remediation is a script that restarts a service when a health check fails. Self-healing is a loop: detect the anomaly, diagnose the root cause, decide on an action, execute it, verify the result, and learn from the outcome. That distinction matters because a DMV-style outage, a network partition or a routing misconfiguration, doesn't respond to a restart. It responds to a rollback, a config change, or a traffic shift. And those actions carry risk.

Your mean time to recovery is too high because humans are the bottleneck. Every incident waits for someone to wake up, log in, triage, and decide. Self-healing agents compress that timeline. But only if you design them to act safely when the stakes are high.

The architecture that holds up

A self-healing agent isn't a monolith. It's four components working together: telemetry ingestion, a policy engine, an action execution plane, and an audit log. Telemetry ingestion has to handle heterogeneous sources: SNMP traps from routers, Prometheus metrics from Kubernetes, syslog from legacy apps. The policy engine evaluates against both static thresholds and learned baselines, but learned baselines introduce false positives if training data includes past incidents. The action execution plane needs an idempotency key and a dry-run mode; without dry-run, a rollback can compound an outage. The audit log must be append-only and tamper-evident, not just a database table. The execution plane is the highest-risk component, which is why sandboxing agent actions before they touch production is non-negotiable.

Click through the five-stage loop—observe, decide, act, verify, learn—and see where human approval gates high-risk actions.

The control loop that ties these components together has five stages: observe, decide, act, verify, learn. Observe means comparing live state to a versioned baseline, not just alerting on threshold breaches. Decide means matching the anomaly signature to a playbook with a confidence score; below 0.8, escalate to human. Act means executing via a limited set of primitives (restart, rollback, scale, drain) with a timeout and rollback-on-failure. Verify means checking not only the original symptom but also dependent service health. Learn means updating the playbook only after a post-incident review, not automatically from a single success.

Bounded autonomy is the principle that makes this safe. Every action an agent can take falls into one of three categories: fully autonomous, approval-required, or forbidden. The category depends on two factors: blast radius and reversibility.

policies:
    - action: restart_container
        blast_radius: single_service
        reversible: true
        approval: none
    - action: rollback_router_config
        blast_radius: multi_service
        reversible: true
        approval: human_required
    - action: delete_database
        blast_radius: catastrophic
        reversible: false
        approval: forbidden

Here's a concrete scenario. A network engineer notices BGP session flapping on a core router. Your agent detects the config drift, compares the live config against the last known-good baseline, and proposes a rollback. But because the blast radius includes multiple services, the agent pauses and waits for approval. The engineer reviews the diff, approves it, and the agent executes the rollback, then verifies the sessions stabilize. That's self-healing with a human in the loop, not instead of one.

Where teams usually fail

The first failure is automating the wrong layer. Restarting a container is safe but rarely fixes a network partition. Teams then jump to automating config rollbacks without a rollback plan for the rollback itself. A common failure is an agent that misdiagnoses a transient latency spike as a full outage and restarts healthy services, causing a thundering herd on startup. Over-automation without a blast radius limit leads to cascading failures: an agent that scales up a database in response to a slow query can exhaust connection limits on downstream services. Missing audit trails are not just a compliance problem; they prevent post-incident learning because you can't reconstruct the decision path. Privilege escalation is the worst case: if the execution plane runs as root, a prompt injection in a log message can trigger a destructive action.

But the most common failure is subtler: teams treat the agent as a black box. Explainability isn't just about trust; it's about debugging. When an agent rolls back a router config, you need the exact diff, the baseline it compared against, and the confidence score. Otherwise you can't tell if the agent fixed the root cause or just masked a symptom. The first agent-induced incident will happen; if you can't explain why, leadership will shut it down. The lesson from AI Agent Failures: Lessons Learned from Enterprise Deployments is that explainability isn't a nice-to-have. It's the difference between an agent that gets better over time and one that gets shut down.

And here's the scenario that keeps platform leads up at night. A database connection pool exhausts during a traffic spike. Your agent scales read replicas and restarts connection pools automatically. That's good. But what if the agent's diagnosis was wrong? What if the real problem was a slow query that the extra replicas just masked? The agent needs to verify the change is reversible and log the decision path before acting. Otherwise you've automated the creation of new incidents.

How to measure progress

Can you measure whether self-healing is actually working? Yes, but not with the metrics you're already tracking. Mean time to recovery alone is misleading because an agent that restarts services aggressively will lower recovery time while raising your change failure rate. You need four signals: mean time to recovery, change failure rate, false positive rate, and agent-induced incident count.

Mean time to recovery tells you how fast problems get fixed. Change failure rate tells you how often fixes break something else. False positive rate tells you how often the agent acts when it shouldn't. Agent-induced incident count tells you how many incidents the agent itself created. If recovery time drops 40% but agent-induced incidents rise from zero to three per quarter, you haven't improved anything. You've just moved the failure point.

For regulated environments, add two more signals: audit completeness and explainability coverage. Audit completeness means every agent action has a timestamped, immutable record that a compliance reviewer can reconstruct. Explainability coverage means the percentage of agent decisions that include a human-readable rationale. If your agent can't explain why it rolled back a router config, a state auditor won't accept the audit trail, no matter how good the outcome was.

What to build next

Start with drift detection, not automation. Before your agent can fix anything, it needs to know what "correct" looks like. Build configuration baselines for your critical infrastructure: routers, load balancers, API gateways, database clusters. Then deploy agents that compare live state against those baselines and flag drift. That's detection without action, and it's the safest possible first step.

Then add failure injection. You can't validate an agent's response to a network partition by waiting for a real partition to happen. Use chaos engineering to simulate the failures you're most afraid of: BGP session flaps, latency spikes, config drift, certificate expiry. The AI Agent Testing Playbook covers the simulation patterns in detail. The goal isn't to prove the agent works. It's to find the failure modes before production does.

And one more scenario worth designing for. An internal API gateway certificate expires. Your agent renews it via ACME, updates dependent service configurations, and verifies end-to-end TLS handshakes before closing the incident. That's the full loop: detect, decide, act, verify, learn. It's also the kind of routine, high-frequency failure that humans are terrible at catching before it becomes an outage. Self-healing agents are at their best here, not in the dramatic hero moments, but in the quiet prevention of the failures that never make the news.

The operating model that emerges is clear. Agents handle the detection, diagnosis, and verification. Humans handle the high-risk decisions and the policy updates. The agent-to-human handoff patterns you build for customer-facing agents apply equally to infrastructure agents. The handoff isn't a failure mode. It's the design.