I run a one-person AI company: Claude Code writes and maintains the code, and most of what it builds runs unattended — several trading bots on Windows Task Scheduler, each polling a broker API every 5 minutes, 24/7, with nobody watching in real time. I wrote before about the agent breaking things it had write access to (I let an AI agent run my trading bots unattended). This one's different: nothing broke the code. The monitoring itself was confidently wrong, twice, in two different ways.
The mild version: a real event, never logged
One bot has a circuit breaker — if cumulative paper losses cross a threshold, it force-closes everything and halts. It fired for real: losses crossed the line, the position closed, confirmed directly against the broker's own API.
Except the close was never written to the trade history file. The reporting script had no record of it. So my daily automated status report — a script that reads every bot's logs and has an AI model summarize what's going on — looked at a "still open" position that had actually been closed for days, and confidently told me the bot might have crashed. It hadn't. It had done exactly what it was supposed to do, and the thing telling me otherwise was itself misreading stale data as current.
Annoying, but honest about being wrong once you dug in. The next one wasn't.
The real version: 18,300 tracebacks, and every single check said "fine"
A different bot's scheduled task kept reporting success — exit code 0, every 5-minute run, for over three weeks straight. The scheduler's own logs showed nothing but green.
Inside, the actual Python process had been crashing on nearly every cycle that whole time: an authentication error from the broker's API, unhandled, caught only by the outer process wrapper, which then dutifully reported "the wrapper ran and exited" as success — which was technically true and completely useless. Three weeks of 5-minute cycles is over 8,000 attempts; more than 18,000 tracebacks piled up in the log because a few different code paths kept trying and kept failing. Zero real trades got recorded in that entire window. Nothing about the scheduler's own view of the world ever turned red.
I only found it because I went and read the raw log file directly, not because anything monitoring the system told me to.
Why the tools I already had didn't (and wouldn't have) caught this
The LLM-observability tools I know of trace individual API calls while you're actively building — good for "why did this one prompt cost so much" or "why did this one call return garbage," not built to watch a background job nobody's looking at.
The classic dead-man's-switch tools (the "ping us every N minutes or we alert you" category) would have shown green the entire three weeks, too — the wrapper process itself never stopped running or stopped pinging. That category answers "did the job run." It has no way to know what the job was actually supposed to accomplish, so it can't tell you the job ran and did nothing.
What both incidents have in common: the failure was invisible to anything that only checks "did the process exit 0" or "did something get logged as a plain string." Neither incident involved the code lying — the wrapper genuinely didn't crash, and the "open position" genuinely had been open at some point. The gap was between "the shell of the job looks fine" and "the job actually did the thing it exists to do."
What I'm taking from this
An agent that's competent while you're watching it, and an agent whose failures you'll actually notice once you stop watching, are not the same property — same lesson as the write-access incidents, different failure shape. Uptime monitoring answers "is it alive." Nobody was asking the more useful question: "is it still doing the thing," specifically for a background AI agent where "the thing" is something more structured than "return HTTP 200."
I'm looking at building a small monitoring layer specifically for solo-developer/small-team unattended AI agents — schedule-aware, understands that "the process exited 0" and "the agent did its job" are different claims, and flags the gap between them instead of only the process dying outright.
If you're running any kind of unattended agent — a scraper, a bot, a pipeline — on a schedule with nobody watching, I'd like to know if you've had your own version of "everything said green and it wasn't." Trying to figure out if this generalizes past my own two data points, same as last time.
Top comments (3)
The part I like here is that the task reported on the wrapper, not the work. I try to make scheduled jobs emit one small domain-level proof at the end, even if it is boring. Row count changed, broker state matched, artifact hash moved, that kind of thing. Exit code alone is too cheap to fake by accident.
Yes, twice, and both had the same shape as yours: the layer that reported success was a real layer, it just wasn't the one doing the work.
I capture screenshots of social posts as evidence for a harassment-preservation service. Every capture came back HTTP 200 with a valid PNG. They were Cloudflare block pages. The wrapper I was going through had gone dead, and a block page is still a successful fetch of a real image - the failure arrived wearing the exact shape of the success.
The other one was last week. My publishing script pushes an article to a git-backed platform. Push succeeds, exit 0. The platform has an undocumented posting-rate limit, and over it the article simply never deploys - the URL sits at 404 and nothing anywhere reports an error. I had already misdiagnosed that once as a frontmatter problem, because a successful push felt like evidence.
What I took from it: the check has to assert the thing you actually want, not a proxy for it. Exit 0 means the wrapper returned. 200 means bytes arrived. Neither is "the work happened." Both of mine are now fixed by asserting the outcome instead - the capture is compared against a known-good stencil, and the publish step polls the article URL for a 200 and refuses to record it as published until it sees one. So it does generalise well past two data points.
The useful split here is exit status versus outcome assertion. A wrapper returning 0 is just a claim about the wrapper. For unattended agents, I would want the monitor to check the traded, published, captured, or reconciled artifact directly, even if that check is slower.