One of my agents exited cleanly at 3 AM, another sat "healthy" while doing zero useful work for four hours, and a third burned through $50 in API credits in 40 minutes without throwing a single error.
Those incidents looked unrelated at first. They weren't. All three slipped past the usual stack of process checks, log watchers, and CPU or memory alerts because those tools were measuring infrastructure symptoms, not whether the agent was still doing useful work.
Failure #1: The Silent Exit
One of my agents exited cleanly at 3 AM. No traceback. No error log. No crash dump. The Python process simply stopped. My log monitoring saw nothing because there was nothing to log.
I found out six hours later when I noticed the bot hadn't posted since 3 AM.
What happened
The OS killed the process for memory. The agent was slowly leaking — a library was caching LLM responses in memory without any eviction policy. RSS grew from 200MB to 4GB over a few days. The OOM killer sent SIGKILL, which leaves no Python traceback.
Why traditional monitoring missed it
- Process monitoring (systemd, supervisor): Saw the exit code, but by the time you check alerts, the damage is done
- Log monitoring (Datadog, CloudWatch): Nothing to see — OOM kill happens below the application layer
- CPU/memory dashboards: Would have caught it if someone was watching. Nobody watches dashboards at 3 AM.
The pattern that catches this
Positive heartbeat. Instead of monitoring for bad signals (errors, crashes), monitor for the absence of a good signal. The agent must actively report "I'm alive" every N seconds. If the heartbeat stops for any reason — clean exit, OOM, segfault, kernel panic — you know immediately.
# Inside your agent's main loop
while True:
result = do_work()
heartbeat() # This is the line that matters
sleep(interval)
If heartbeat() doesn't fire, something is wrong. You don't need to know what — you need to know when.
Failure #2: The Zombie Agent
This one is more insidious. The process was running. CPU usage normal. Memory stable. Every health check said "healthy."
But the agent hadn't done useful work in four hours.
What happened
The agent was stuck on an HTTP request. An upstream API had rotated its TLS certificate, and the request was hanging — the socket was open, the connection was established, but the TLS handshake was deadlocked. No timeout was set on the request (a classic oversight).
From the outside, the process was "running." From the inside, the main loop was blocked on line 47 of api_client.py, and it would stay blocked forever.
Why traditional monitoring missed it
- PID checks: Process exists ✓
- Port checks: Agent's HTTP server responds ✓ (the health endpoint runs on a separate thread)
- CPU/memory: Normal ✓
The health check thread was fine. The work thread was dead.
The pattern that catches this
Work-progress heartbeat. A background-thread heartbeat (like the one in Failure #1) catches crashes and OOM kills — it proves the process is alive. But it can't catch zombies, because the health-check thread keeps running even when the work loop is stuck.
For zombie detection, the heartbeat must come from inside the work loop:
# Level 1 — Liveness (background thread)
# Catches: crashes, OOM kills, clean exits
# Misses: zombies, hung calls, deadlocks
threading.Thread(target=lambda: while True: heartbeat(); sleep(30)).start()
# Level 2 — Work-progress (inside the loop)
# Catches: everything above + zombies, hung API calls, logic deadlocks
while True:
data = fetch_from_api() # If this hangs...
process(data)
heartbeat() # ...this never fires
sleep(interval)
Both levels are valid — they answer different questions. A background thread measures "is the process alive?" A work-loop heartbeat measures "is the agent making progress?" For full coverage, you want both.
Try it now — monitor your agent in 2 lines:
pip install clevagent
import clevagent
clevagent.init(api_key="***", agent="my-agent")
Free for 3 agents. No credit card required. Get your API key →
Failure #3: The Runaway Loop
This is the scariest failure mode because the agent looks great. It's running. It's doing work. It's calling the LLM API, getting responses, processing them, and calling again. Every metric says "healthy."
Except your bill is exploding.
What happened
The agent received a malformed response from an API. It asked the LLM to parse it. The LLM returned a structured output that triggered the same code path again. The agent asked the LLM to re-parse. Same result. Repeat.
Token usage went from 200/min (normal) to 40,000/min. In 40 minutes, it burned through about $50 of API credits. Not catastrophic for a single incident, but imagine this happening overnight with a larger model.
Why traditional monitoring missed it
- Process health: Running ✓
- Heartbeat: Firing normally ✓ (the loop is running, just wastefully)
- Error rate: Zero ✓ (no errors — the LLM is responding successfully every time)
- CPU/memory: Normal ✓ (LLM calls are I/O-bound, not compute-bound)
The pattern that catches this
Cost as a health metric. Track token usage (or API cost) per heartbeat cycle. If it spikes 10-100x above baseline, flag it.
while True:
start_tokens = get_token_count()
result = do_llm_work()
end_tokens = get_token_count()
heartbeat(
tokens_used=end_tokens - start_tokens,
cost_estimate=calculate_cost(end_tokens - start_tokens)
)
sleep(interval)
This is the one metric that's unique to LLM-backed agents. Traditional services don't have a per-request cost that can spike 200x. AI agents do.
The Monitoring Stack for AI Agents
After dealing with all three failures, I realized the monitoring requirements for AI agents are fundamentally different from web services:
| What to monitor | Web service | AI agent |
|---|---|---|
| Is it alive? | Process check | Positive heartbeat (agent must prove it's alive) |
| Is it working? | Request latency | Application-level heartbeat (from inside the work loop) |
| Is it healthy? | Error rate | Cost per cycle (token usage as health signal) |
The minimum viable version of this is surprisingly simple:
- Put a heartbeat call inside your main loop (not in a health-check thread)
- Include token/cost data in each heartbeat
- Alert on silence (missed heartbeat) and on cost spikes
That alone would have caught all three of my failures within 60 seconds instead of hours.
Where ClevAgent fits
If you do not want to wire this yourself, ClevAgent packages the same operating pattern: heartbeat freshness, loop and cost-spike detection, auto-restart, and daily reporting for long-running agents.
But the pattern matters more than the product mention here. Even if you roll your own with a webhook plus PagerDuty, the three signals above — heartbeat, work-progress freshness, and cost tracking — will catch most of the failures that basic infra monitoring misses.
The dangerous cases are not just crashes. They are the hours where the process still looks alive while useful work has stopped or spend has detached from baseline. If you want a runtime watchdog built around those signals, start monitoring with ClevAgent.
Start monitoring your agents for free
ClevAgent is free for up to 3 agents — no credit card required. Add one line to your agent loop and get heartbeat monitoring, zombie detection, runaway cost alerts, and auto-restart in minutes.
Top comments (0)