262 Nights, Zero Bytes, Exit 0: The Failure Mode of Unattended Agent Runs

A backup job I run synced an empty directory every night for 262 consecutive nights. Every run exited 0. Every run wrote Photo library sync complete into the log. The off-site copy held zero objects for nine months while the real library sat at 17 GB and 17,658 files on a disk nothing was copying.

The script's source path pointed somewhere that did not exist. Docker's bind mount then created that path, empty, at the start of every run. The job manufactured the evidence that its own configuration was valid, found nothing there, and synced the nothing it found. A guard like [ -d "$source" ] would have passed all 262 times.

I found it by counting objects in the bucket by hand, on an unrelated errand. No monitor caught it, because there was nothing for a monitor to catch. The job ran on schedule. It logged. It exited clean.

That job is what I think about now when a vendor shows me an agent that works unsupervised for two days.

An exit code answers a different question than the one you asked

AWS now sells what it calls frontier agents: autonomous systems that, in its own words, "run persistently for hours or days without constant human oversight". The Security Agent bills $50.00 per task-hour, and AWS puts an average evaluation at 24 task-hours, so roughly $1,200 a run. I wrote about that pricing model in April and I still think the unit economics are sound.

The economics are not the problem. Supervision is.

When a run ends, what comes back is a status: it finished, or it didn't. One bit. That bit reports whether the process reached its final line, which is a different question from whether the work happened. My backup reached its final line 262 times.

For a five-minute job you paper over the gap by looking. You read the output, you spot-check the result, your own attention is the instrument. Stretch the same job to 24 unattended hours and that instrument is gone. What's left is the bit.

Success decays exponentially with run length

A 24-hour run isn't a stretched-out 1-hour run. The failure math works against you, and it compounds.

Toby Ord's analysis of agent benchmark data, Is there a half-life for the success rates of AI agents?, fits the results to a constant rate of failure per minute of work. That produces exponential decay: success probability falls off a cliff as tasks get longer, and every agent has a characteristic half-life. Failures accumulate across subtasks, and one bad subtask sinks the run.

METR measures the same curve empirically. Its time-horizons work reports the task length at which frontier models succeed 50% of the time, and that 50% threshold is chosen because it's the easiest point to estimate robustly. At the horizon being advertised, the coin is fair.

Then read the caveat on METR's own page. "Measurements above 16 hrs are unreliable with our current task suite."

The independent yardstick runs out at 16 hours. The vendor's advertised operating range is hours or days. Past that line there is no public, non-vendor measurement of how often these runs actually succeed. You are not choosing between a measured 94% and a measured 60%. You're choosing between a vendor number and no number.

My own agent measurements lied in the same direction

This isn't a hypothetical I imported from a paper. My own eval tooling has handed me clean results that were nothing of the kind.

A model evaluation I ran logged a 900-second timeout and a 766-second near-timeout for two coding tasks. The CSV was indistinguishable from a model that cannot do the work. A second agent had been pushing large prompts at the same single-slot inference server the whole time. Re-run on a quiet box, the same model and the same tasks passed in 57 and 83 seconds.

A separate eval launched into a detached tmux session returned two FAIL rows in under a second, with zero-byte logs. The binary wasn't on the PATH that a daemon-spawned shell inherits. That output is indistinguishable from a model that errors instantly.

Both runs produced a result file. Both exited. Neither measured what it claimed to measure, and in both cases the artifact looked exactly like a real finding, which is the part that costs you. A crash you notice. A plausible wrong answer goes into the spreadsheet.

The same pattern shows up wherever I stopped watching. A weekly link checker reported Scanned 0 wiki pages / Dead wikilinks: 0 for six weeks, green every time, because it pointed at a directory a migration had retired. Repointed, the same code immediately found 261 pages and 116 dead links. A watchdog probing an embedding server's /api/version endpoint stayed green for two days while every actual embed call returned a 500 and the index took in nothing.

A 94% accuracy score is computed over the runs somebody checked

The strongest objection is that this is all anecdote against data. AWS reports up to 75% lower MTTR, 80% faster investigations, and 94% root cause accuracy from preview customers, and one university cut incident resolution from two hours to 28 minutes. Those are outcomes, not vibes.

Look at how that number gets built. A root-cause accuracy figure is computed over runs whose root cause somebody established independently. To score a run, you need ground truth, which means a human went and checked. The denominator is runs that were verified.

The failure class I'm describing never enters that denominator. A run that reports success, produces a clean artifact, and is wrong is scored as correct unless somebody goes looking, and the whole premise of an unattended agent is that nobody does. My backup would have scored 262 for 262 on any accuracy metric derived from its own output.

These are also first-party numbers from preview customers selected by the vendor. That's not disqualifying. It does mean they're a ceiling rather than an expectation, and they answer "how often is a checked run right," not "how often does an unchecked run lie."

Assert the artifact, not the exit status

The fix isn't more monitoring. It's changing what gets asserted. The backup is the shortest example I have:

# Before: the job's own bind mount created $SRC empty, so this always passed.
[ -d "$SRC" ] || { echo "missing source"; exit 1; }
rclone sync "$SRC" "$DEST"

# After: refuse an empty source, and log what actually moved.
[ -n "$(find "$SRC" -type f -print -quit)" ] || {
  echo "REFUSING: $SRC is empty — not syncing a deletion"; exit 1; }
rclone sync "$SRC" "$DEST"
rclone size "$DEST"   # object count and bytes, into the log, every single run

Refuse a degenerate input instead of processing it. rclone sync mirrors deletions, so a wrong path doesn't merely fail to back things up, it erases the copy you had. Emit a number that a zero-work run cannot produce. And push the heartbeat only when a leg moved a non-empty source, so a silent no-op goes stale and pages somebody.

For an agent run the shape is identical. Don't ask whether the run finished. Ask what it changed: how many findings, how many files touched, how many tests that were red are now green, how many tokens against how many accepted diffs. Then set a floor. A 24-hour pen test that reports zero findings and zero requests issued has told you about your instrumentation, not your security posture.

The cheap version of this test: can this check ever come back red? If a run that did nothing would produce output identical to a run that worked, you don't have a check. You have a log line.

Where this doesn't apply

Exploratory work has no assertable artifact. When you turn an agent loose on "look at this system and tell me what's odd," there's no count to floor and no diff to measure, and you're back to reading the transcript. Outcome assertions don't help there, and pretending otherwise gives you a number that means nothing.

The threshold is unattended duration, not autonomy. A 20-minute agent run you watch needs none of this. An hour-long cron job nobody has opened since 2025 needs all of it. Agents make it urgent because the unattended part is the product, sold at durations past where anyone outside the vendor has measured the success rate.

262 nights is the number I'd put in front of anyone signing off on a Friday-to-Monday agent run. Not because a backup script is an agent, but because it's the floor: a deterministic shell script, no model in the loop, nothing nondeterministic about it, wrong every night for nine months while reporting success. Add nondeterminism and a 50% time horizon on top of that and the exit code is worth less, not more.

Decide what a successful run must produce, in a number, before you let anything run for a day. Otherwise you'll do what I did, and find out by hand, on an errand, in month nine.