My OpenClaw agent crashed three Sundays ago. Not a slow degradation — it just stopped. The cron job that runs every morning to prep my calendar fired, couldn't reach the calendar API, and instead of retrying or falling back, it logged a vague error and quit. I didn't notice until 10 AM, when I was standing in line for a meeting I should have been in 20 minutes early for.
That meeting cost me something. Not a lot. But enough that I spent the next week building systems into my agent that would have caught it — and more importantly, fixed it — without me.
This is what I learned about building self-healing into an AI agent. Not abstract principles. Real patterns that run on my machine right now.
Pattern 1: The Heartbeat You Actually Check
Most monitoring is theater. You set up a cron to alert you when something breaks, but if that cron is the thing breaking, you're still blind.
OpenClaw's heartbeat system runs on a configurable interval and tracks whether your agent is actually responsive. The default configuration is conservative — it waits a long time before declaring something wrong. I tightened mine to trigger a recovery check within 5 minutes of a missed heartbeat.
Here's the key: the heartbeat isn't just a ping. It's a state file. When my agent runs, it writes its current status to a JSON file. When the heartbeat fires, it reads that file. If the timestamp is stale, it knows the agent went dark — and can trigger a restart before I ever get a Telegram alert.
// heartbeat-state.json (written by agent on every significant action)
{
"lastAction": "cron-daily-planning",
"timestamp": "2026-07-28T06:45:12Z",
"sessionActive": true,
"errors": []
}
The beauty is this: the agent writes the state. The heartbeat monitors it. The watchdog restarts it. Three separate systems that can't all fail at once without something being seriously wrong.
Pattern 2: The Dreaming Sweep — Memory That Repairs Itself
Every night at 2 AM, my agent runs a "dreaming" process. It reviews the last 24 hours of its own recall entries — every tool call, every decision, every pattern that showed up repeatedly. It looks for signals in the noise.
Most agents have long-term memory. Mine has a curated long-term memory. The dreaming sweep stages candidates, scores them, and only promotes entries that appear in at least 3 separate queries across 3 different sessions with a minimum quality score of 0.8. Everything else gets discarded.
This sounds like overkill. It isn't.
Six months ago, my agent was polluting its own memory with single-occurrence noise. Every odd query, every experimental tool call, every failed attempt — all of it getting stored and weighted equally. When it tried to recall something actually important, it was buried under a hundred irrelevant one-offs.
The dreaming sweep fixed that. Memory in my agent now has a half-life. Things that don't recur get quietly forgotten. Things that matter — because they keep mattering — compound.
Pattern 3: The Self-Check That Doesn't Lie to Itself
Here's a failure mode I didn't anticipate: my agent was validating its own work using criteria it had written. Which meant it was grading its own answers using rules it knew how to game.
I'd ask it to review a document for errors. It would find the errors, fix them, and then rate its own fix at 95% confidence. Was that accurate? Who knew. It was certainly confident.
I killed the self-check loop and replaced it with a two-step verification: the agent does the work, then a separate isolated sub-agent reviews it using a different model and a different prompt structure. No access to the original output. Just the original task.
The isolated agent can't know what the first agent said. It has to evaluate independently. If they agree, the confidence is real. If they disagree, I get the disagreement, not a false consensus.
This sounds expensive. It is slightly. But I've caught six silent failures in three days using this setup. Each one was something that would have shipped if I'd trusted the original agent's self-assessment.
Pattern 4: The Cron Chain That Doesn't Chain
Standard cron jobs fail silently. You set them, you forget them, and six months later you discover they stopped working three Tuesdays ago and nobody noticed.
My OpenClaw crons don't chain — they cascade. If the morning planning cron fails, a watchdog timer notices within 10 minutes. It doesn't restart the same cron. It runs a diagnostic first: checks what API was unreachable, checks current connectivity, and either retries or routes around the failure by using a cached fallback.
If the calendar API is down, my agent switches to a plain-text reminder file it maintains. When the API comes back, it reconciles. I don't get calendar notifications that morning — but I get something instead of nothing.
# watchdog timer — triggers if main cron hangs
[Unit]
Description=OpenClaw Cron Watchdog
[Timer]
OnBootSec=5min
OnUnitActiveSec=10min
Unit=subagent-watchdog.service
The systemd timer watches the watchdog. If the watchdog hangs, something is seriously wrong at the OS level — and I get an alert for that too.
What This Actually Means
Building self-healing into an agent isn't a feature you install. It's a posture. You have to assume things will break — and not just the things you can anticipate. The 3 AM failure wasn't the calendar API going down. It was the cron job not having a retry strategy, a fallback output, or a watchdog to catch the silence.
Once I started treating my agent like production infrastructure instead of a helpful script, the failures became instructive instead of costly. Every break taught me something the agent needed to survive the next break.
The agent isn't autonomous yet. But it's getting better at staying alive long enough to ask me questions when it actually needs to.
What I learned: self-healing isn't a setting you enable. It's a design philosophy you build into every layer — monitoring, memory, verification, and fallback. Start with the heartbeat. Everything else builds from knowing when something goes dark.
Top comments (0)