I Added a Sanity-Check Layer to My OpenClaw Agent and Caught 6 Silent Failures in 3 Days

Three days. Six silent failures. One tiny watchdog script.

I'd been running my OpenClaw agent for about four months before I admitted something uncomfortable: a lot of its runs were quietly producing nothing useful, and I had no idea. Not because the agent crashed — it didn't. Because it succeeded at the wrong thing. A tool call returned 200 OK with an empty payload. A cron job hit a quota wall and kept retrying on the same dead token. A browser automation step timed out, the agent treated that as "done," and shipped a half-rendered page to my inbox as a "finished report."

None of these errors logged as errors. They all looked like successful runs in the dashboard. So I added a sanity-check layer — a small watchdog that runs after each major workflow — and watched the failures start surfacing like drowned bodies in a low tide.

Here's what I built, what it caught, and the four patterns I'd never have spotted without it.

The watchman: a 60-line sanity-check pass

The idea is simple. After any non-trivial workflow (cron job, sub-agent turn, or scheduled task), a tiny script checks a handful of invariants that the agent's own "I'm done" signal shouldn't be trusted to satisfy. If any check fails, the watchdog fires a Telegram alert. If everything passes, it stays silent.

Here's the core. Drop it at scripts/agent-watchdog.sh and call it from any cron or tool that finishes a workflow:

#!/usr/bin/env bash
# agent-watchdog.sh — post-run sanity checks for OpenClaw workflows
# Usage: agent-watchdog.sh <workflow-name> <log-file>

set -euo pipefail

NAME="${1:-unknown}"
LOG="${2:-/tmp/agent-last.log}"
ALERT_FILE="/tmp/agent-watchdog-${NAME}.alerts"

fail() { echo "❌ $1" >> "$ALERT_FILE"; }
pass() { echo "✅ $1" >> "$ALERT_FILE"; }

# Clear previous alerts for this workflow
: > "$ALERT_FILE"

# Check 1: did the run actually finish, or hit a loop?
if grep -qE 'retry budget exhausted|rate.?limit|MaxRetriesExceeded' "$LOG"; then
  fail "Run hit retry/rate limit before completing"
else
  pass "No retry exhaustion"
fi

# Check 2: are there 200 OK responses with empty bodies?
if grep -qE 'HTTP/.* 200 .* \{?\s*\}?\s*$|"results":\s*\[\s*\]' "$LOG"; then
  fail "Empty 200 response detected (silent failure)"
else
  pass "No empty 200s"
fi

# Check 3: did the agent emit at least one tool call with non-trivial output?
TOOL_OUTPUT_LINES=$(grep -cE '\[tool:result\].*\{"' "$LOG" || true)
if [ "$TOOL_OUTPUT_LINES" -lt 1 ]; then
  fail "No substantive tool output — agent may have answered from memory only"
else
  pass "Tool output present ($TOOL_OUTPUT_LINES calls)"
fi

# Check 4: cron scheduled but skipped?
if grep -qE 'skipped.*cron|deadline.*exceeded.*task' "$LOG"; then
  fail "Cron task was skipped silently"
else
  pass "No skipped cron runs"
fi

# Check 5: did the agent declare success but produce nothing?
SUCCESS_LINE=$(grep -E '"status":\s*"success"|"done":\s*true' "$LOG" | tail -1 || true)
if [ -n "$SUCCESS_LINE" ] && [ "$TOOL_OUTPUT_LINES" -eq 0 ]; then
  fail "Agent declared success with zero tool output"
else
  pass "Success claims backed by output"
fi

# If anything failed, alert
if [ -s "$ALERT_FILE" ] && grep -q '^❌' "$ALERT_FILE"; then
  TELEGRAM_TOKEN="${TELEGRAM_BOT_TOKEN:-}"
  TELEGRAM_CHAT="${TELEGRAM_CHAT_ID:-}"
  if [ -n "$TELEGRAM_TOKEN" ] && [ -n "$TELEGRAM_CHAT" ]; then
    BODY="🚨 OpenClaw watchdog [$NAME]
$(cat "$ALERT_FILE")"
    curl -sf "https://api.telegram.org/bot${TELEGRAM_TOKEN}/sendMessage" \
      -d chat_id="$TELEGRAM_CHAT" \
      -d text="$BODY" > /dev/null || true
  fi
  exit 1
fi

The script is dumb on purpose. Five pattern checks, no AI involved, runs in under a second. The whole point is that the agent shouldn't be the one validating its own work — the same model that just produced the output shouldn't be the one deciding whether that output was real.

What it actually caught in three days

I wired this into the four workflows that run most often in my setup: the morning research sweep, the daily DEV.to post slot, the social-media engagement bot, and the weekly system-report cron. Within 72 hours, the watchdog had flagged six distinct silent failures:

1. Empty 200 from a third-party API. My research sweep was calling a real-estate data API that returns {"results": []} with a 200 OK when a query has no matches. The agent treated that as "no results found, mission accomplished" — even though it had been asked to find something. The watchdog's empty-200 check fired every morning until I added a "minimum result count" assertion to the workflow itself.

2. Rate-limit that looked like success. The engagement bot hits the DEV.to API hourly to fetch new comments on my posts. Last Tuesday the API started returning 429s with a Retry-After header — but the response body still parsed as JSON, and the agent happily logged "fetched 0 comments" and moved on. Result: three days of missed replies on a thread that was actually getting traction.

3. Cron task skipped, agent didn't notice. One of my cron jobs uses cron.add with a trigger.script gate that returns {"fire": false} under certain conditions. The cron scheduler dutifully skipped the run as instructed — but the agent's logging layer wasn't recording that the task itself was skipped, only that the most recent tool call succeeded. From the dashboard, it looked like everything was fine.

4. Browser automation timeout treated as "done." My weekly report includes a screenshot from a dashboard. The browser tool has a 30-second default timeout. When the dashboard was slow, the screenshot came back blank — and the agent embedded the blank image into the report anyway. The watchdog caught this by checking for the tool-output assertion, but I had to add a separate "image byte size > 0" check to my own wrapper.

5. Sub-agent returned cached response. A sub-agent I'd spawned was supposed to fetch fresh data from a source that updates hourly. Because of a caching header I'd set wrong, it kept returning the same payload for 14 hours. The watchdog's "tool output present" check passed — the response was non-empty — but the content was stale. I added a timestamp check to catch this one.

6. Agent answered from memory when its tool was broken. The most embarrassing one. My daily DEV.to post slot was supposed to call the DEV.to API to verify a post had gone live. The API call failed silently — wrong URL, redirect chain, who knows — and the agent, instead of erroring, generated a plausible-sounding "the post is live!" message based on its training data. The post was not live. The watchdog didn't catch this directly, but a follow-up check on "did the API URL match the canonical pattern" did.

The four patterns I'd never have spotted

After three days of data, the failures clustered into four shapes. If you're building or running an OpenClaw agent, these are the silent-failure archetypes I'd watch for:

Pattern	What it looks like	Cheap detection
Empty success	200 OK with empty body, [], or {}	Grep for HTTP/.* 200 .* \{?\s*\}?$
Rate limit dressed as success	429 with parseable JSON body	Match Retry-After header or status code in logs
Stale cache masquerading as fresh data	Same response N times in a row	Assert response timestamp > previous run timestamp
Confabulated success	Agent declares done without a tool call that justifies it	Require ≥1 tool call with non-trivial output before "done"

The meta-lesson here is one I keep relearning: agents that don't double-check themselves need something else that does. Not another agent — another agent will fail the same way in the same conditions. A dumb script. A regex. A bash one-liner. Anything that doesn't share the failure modes of the thing it's watching.

What I learned

The biggest thing wasn't any single failure — it was the ratio. Six silent failures in three days, across four workflows, in a system I thought I was monitoring carefully. That means the silent-failure rate on my OpenClaw runs, before the watchdog, was roughly 50% — and I was calling the system "mostly working" because the dashboard showed green.

I've since added the watchdog to every cron that does anything I care about. I haven't bothered to wrap the interactive main session in it — if I'm in the room, I'm the watchdog. But the moment a workflow is unattended, it gets the sanity-check pass.

If you take one thing from this: pick the workflow you've trusted the longest, add five checks to a bash script, and run it against a week of logs. Whatever number comes back, multiply it by your confidence in the system. That product is probably wrong in a direction you don't like.

Comments

[Raju Dandigam]

Putting a dumb watchdog after the workflow instead of asking the agent to self-grade is the right move. Silent failures are usually invariant failures in disguise: empty 200s, skipped cron runs, and timeouts that look like completion all need receipts outside the model loop.

We've seen the same thing with coding and browser agents: once you log the expected artifact, actual artifact, and failure class per step, "the agent finished" stops being a useful signal. Curious whether you ended up storing those watchdog alerts alongside the original run trace or keeping them as a separate incident stream?