A scheduled agent that crashes is the cheap failure. You get a stack trace, a non-zero exit code, a red run in the dashboard, and you fix it that afternoon. The expensive failure is the one where the cron fires on time, the process runs for 40 seconds, exits 0, and does nothing at all. Nobody notices for two weeks.
We run several cron-driven agents in production: one that pulls topic candidates from public feeds, one that drafts and queues articles, one that fans out syndication to three separate networks. Each of them has failed silently at least once. None of those failures produced an error. Here is what actually broke, and what the instrumentation looks like now.
Exit code 0 means the code ran, not that the work happened
The default success signal for anything cron-driven is "the process terminated normally." That signal is close to worthless for agents, because an agent's job is conditional by design: read some source, decide whether there is work, do the work. A clean exit is indistinguishable from "there was nothing to do," which is itself indistinguishable from "the source lied and said there was nothing to do."
We found this the hard way with a discovery job that reads a public listings feed. The feed switched from returning JSON to returning an HTML interstitial for unauthenticated clients. Our parser did what parsers do — it found zero matching items, returned an empty array, and the agent logged 0 new candidates and exited 0. That log line had appeared on plenty of legitimately quiet days, so it read as normal. Eleven days of runs later, someone asked why the topic queue had not moved.
The general shape: any failure that maps cleanly onto a valid empty result is invisible. Most upstream degradations do exactly that.
The four modes that never throw
Silent auth degradation. Expired credentials rarely produce a clean 401 in the wild. A public procurement portal we poll started returning 200 with a login page body once the session cookie aged out. A freelance marketplace's API returned 200 with an empty results array for a revoked token. Both are indistinguishable from "no new records" unless you assert on something other than the status code.
Swallowed errors inside a fan-out. Our syndication step posts to three networks and is deliberately failure-tolerant, so one dead network does not block the others. That is the right design, and it is also how one channel stopped receiving posts across 57 articles: each per-channel error was caught, logged at info level, and the parent step still reported success. Failure tolerance without per-branch accounting is failure concealment.
The schedule stops firing. This one produces no logs at all, which makes it the hardest to spot — you cannot alert on a log line that never gets written. Causes we have hit: a container redeploy that dropped the crontab, a runner quota that silently skipped queued jobs, and a DST shift that moved a 02:30 job into an hour that did not exist that night.
Model output that parses but is empty. An LLM step that returns well-formed JSON with a blank body field, or three bullet points that all restate the title, sails through schema validation. The pipeline continues, writes the artifact, and the failure only surfaces on a rendered page days later.
The
try { ... } catch (e) { log.info(e) }pattern you added to make a nightly job "resilient" is the most common cause of a silently dead agent we have run into. Resilience means the run continues and the failure is counted. If a caught exception does not increment something you alert on, you have not made the job resilient — you have made it quiet.
Assert on the artifact, not on the run
The fix that mattered most was changing what counts as evidence. A run's own report of itself is not evidence; the thing it was supposed to produce is.
Three checks cover most of it:
| Check | What it catches | Where it lives |
|---|---|---|
| Freshness assertion on the output | Empty results, auth degradation, schedule stopped firing | Separate job, separate schedule |
| Per-branch success counters | Swallowed errors inside a fan-out | Inside the agent |
| Shape assertions on model output | Parseable-but-empty generations | Inside the agent, before write |
The freshness assertion catches the widest class, because it is defined entirely in terms of the world rather than the job. Ours is roughly: if the newest row in the candidates table is older than 36 hours, alert. That single check would have caught the HTML-interstitial failure on day two instead of day eleven, and it also catches a cron that stopped firing, which no amount of in-process instrumentation can.
Run it from somewhere the agent cannot take down with it. A check that lives in the same cron file as the job it watches will go missing at exactly the moment you need it.
For per-branch counting, we stopped reporting a boolean and started reporting a tuple: attempted, succeeded, skipped-with-reason. A run where attempted is 3 and succeeded is 2 is a passing run with a warning, not a green check. That distinction sounds pedantic until you weigh it against 57 articles that were never announced anywhere.
Shape assertions are cheap and worth writing even when they feel redundant. Ours reject a generated block if any field falls under a minimum character count, if two items are more than 80% similar to each other, or if the output repeats the input title verbatim. They fire maybe once every few dozen runs — often enough to justify twenty lines.
Give every scheduled agent a deliberate no-op path that logs distinctly.
checked 412 items, 0 matchedandchecked 0 itemsare very different events. Collapsing both into "nothing to do" is what makes the failure invisible.
Dry-run before you schedule
Before a scheduled agent goes into cron, run it interactively against production credentials with writes disabled, and read the whole transcript. Most of the modes above are obvious in a transcript and invisible in a log aggregator — the HTML interstitial is right there in the response body, and no log line was ever going to show it to you.
Doing this in a terminal agent that keeps the session and intermediate state open, so you can inspect a parsed response without re-running the entire job, cut our time-to-diagnosis on this class of bug more than any dashboard did.
Then set the freshness alert before the first scheduled run, not after the first incident. The alert is not overhead you add once the agent has proven itself — it is the only thing that will tell you whether the agent is working at all.
Originally published at pickuma.com. Subscribe to the RSS or follow @pickuma.bsky.social for new reviews.
Top comments (0)