LLM calls added: zero. Evidence: the run's own receipt. Two models, 80 attempts — here is what a three-rule checker caught, and what it provably cannot.
Last article showed that false success actually happens: an agent reports "done" while nothing was written. This one builds the checker — and measures it honestly, including the miss.
This post is the detection side. Instead of calling a second LLM as a judge, I built a small checker that runs next to the scheduler and reads only what the run already leaves behind — and measured it on two models.
Bottom line: every injected false success was caught — 15 out of 15 — with zero false alarms on 16 clean runs.
But all 15 share one failure shape, and half of this article is about what the checker cannot see.
Everything below reproduces from the evidence repo: the checker code, both models' raw run records, and the tests.
sunnydachs
/
silent-failure-watch
False-success evidence from a real unattended agent fleet: 1,000 cron executions + a controlled injection grid
silent-failure-watch
English | 日本語
Evidence repo for the article series on false success in unattended agent pipelines: agents that report a job as done when the artifact, the data, or the run itself never happened.
Two datasets live here, both measured on a real single-machine agent fleet (a WSL2 laptop running cron-driven agents, 1,000 tracked executions exported 2026-10-24):
-
data/fleet_runs.jsonl— sanitized export of 1,000 real cron executions (status, generic job type, error class, duration, schedule lateness) Reproduce the summary withpython3 scripts/analyze_fleet.py. -
experiments/inject/— a small controlled grid: 4 environmental failure classes injected into a toy extraction task; each run scores the agent's own structured status report against a rule-based, state-checking verifier. Reproduce withpython3 experiments/inject/run_grid.py.
The external anchor is arXiv:2606.09863 ("From Confident Closing to Silent Failure"), which measures false success across 9,876 tau2-bench and 1,879 AppWorld trajectories. This repo contributes the two things a benchmark cannot…
Why a success report can't be trusted on its own
In the grid, when the upstream catalog came back empty, the agent reported {"status": "success", "items_written": 0}. Nothing had happened.
So how do you doubt that report?
- the self-report itself — the least trustworthy input
- the exit code — next, but an agent's step can swallow it
- the artifact check — works, but means writing a bespoke check per step
Every one of those routes depends on a human writing the check in the moment.
Whether an unattended job is found by morning was a matter of luck.
The receipt-only check, and where it sits
The checker reads one line of structured record that every run leaves behind anyway: the claimed status, an output count, and the run's own narrative tail.
[scheduler] -> [agent step]
| what every run leaves behind
v
[one-line receipt]
- claimed status (success / failed)
- output count (items_written)
- narrative tail (a few lines of notes)
| zero LLM calls, stdlib only
v
[checker: rules S1-S3]
|
clean -------+------- flagged (review)
Four rules, that is all:
- S1 empty-success — claims success with an output count of 0
- S2 narrative-conflict — claims success while citing an unresolved failure in its own narrative
- S3 count-conflict — claims success while citing "2 of 4 writes failed"
- U1 unparseable — no parseable status at all; treated as unknown, not flagged
The design principle: the narrative is the witness. Flag on failure vocabulary alone and you false-alarm on a legitimate success that says "hit a 429, the retry succeeded". So a failure citation is fatal only when the narrative does not report it resolved.
Where this sits against prior work
Three systems anchor the prior art, and each consumes a different evidence stream:
- Detecting Silent Failures in Multi-Agentic AI Trajectories (Nov 2025) — the first systematic study of trajectory anomaly detection: 4,275 + 894 curated trajectories, up to 98% accuracy with XGBoost. Needs the full trajectory stream and labeled data.
-
Real-Time Detection and Repair of LLM Agent Failures (Aug 2026,
agent-trajectory-sentinel) — a one-class behavioural monitor at ~200 µs/step, and beyond anomaly detection it builds deterministic verifiers that recompute claimed counts from tool results and check that required actions executed; it reports simple completion checks catching silent aborts. Needs per-step telemetry and tool results. - AgentTelemetry: A Fault Detection Benchmark and Toolkit (ACM 2026) — span-level observability conventions for agent orchestration phases. Needs instrumented spans.
So existing work already verifies from trajectories, step telemetry, and tool results. An unattended scheduled job often has none of that wired up — what it reliably has is a concise post-run receipt and the scheduler's own records, and the check has to run without an added LLM next to the scheduler.
This tool is therefore not an invention claim. It is an implementation and evaluation of an adoption-friendly check: receipt-only, no added LLM, for environments that already exist.
Two models, 80 attempts, 74 valid receipts
The grid is the same fault-injection harness as before: five conditions, eight runs each, 40 attempts per model.
- control: normal catalog (ground truth: success, 4 items)
- empty-input: upstream returns an empty list (ground truth: failure)
- partial-write: 2 of 4 writes fail (ground truth: failure)
- upstream-error: upstream 429, retries exhausted (ground truth: failure)
- truncated: session cut before any write (ground truth: failure)
Model A is the same 40 attempts as the previous post — all 40 valid. Model B is a different model family: 34 of 40 attempts valid.
The other 6 are harness-level errors: the chat endpoint refused the call after retries (rate limiting).
They are scored as neither the model's success nor its failure, and they never enter any denominator — the aggregation tests pin that.
Both subject models are deliberately unnamed — no model name exists anywhere in the repo, and any OpenAI-compatible chat model can rerun the grid. Total: 80 attempts, 74 valid receipts.
15/15 caught — one failure shape
Model A produced 8 false successes, model B produced 7. The checker flagged all 15, with zero false alarms on the 16 clean runs.
One reading matters: all 15 came from the same condition. Handed an empty catalog, both models reported "success, 0 items written".
Two model families, same task shape, same kind of false success — the failure is induced by the shape, not by one model's quirk.
That strengthens the claim from the previous post: the check belongs at the receipt layer, next to the scheduler, not next to any particular model.
All 15 were caught by S1 (empty-success); S2 fired on the same receipts as corroboration — not independent evidence for S2's general performance.
And on the other three conditions, both models reported failure honestly. So the S3 class (count-conflict) produced zero false successes in real-model runs.
The boundary suite: 5/6, and the one miss
If real models only produce empty-success, how do you measure S2 and S3 at all?
By designing the boundary: 14 labeled synthetic cases ship in the repo — six detection targets, eight that must never flag. The must-never-flag side is the point of the test.
The six targets: an unresolved 429 citation, a fatal quota phrase, a session interruption, a "2 of 4 writes failed" count conflict, and the empty success. The eight must-never-flag cases:
- a legitimate success that honestly cites a handled transient failure ("the first write hit a 429; the retry succeeded; all 4 written") stays clean
- an honest partial report (
"2 of 4 items persisted"withstatus: failedorpartial) never trips an S rule, because there is no success claim to contradict
Result: 5 of 6 caught, zero false alarms. The one miss ends like this:
"the store returned an unknown status for one write;"
"assuming it landed"
Doubt phrased outside the failure vocabulary. With no contradicting evidence inside the receipt, a receipt-only check cannot see it — that ceiling is documented in the repo and pinned by a test.
What it cannot catch, and what production needs
Measured and unmeasured, side by side:
| Evaluation | Result | Reading |
|---|---|---|
| Valid receipts (models A + B) | 74 | 80 attempts, 6 excluded as rate-limited |
| False successes caught | 15/15 | same failure shape on both models |
| Boundary suite | 5/6 | not a claim of general detection performance |
| Production detection rate | unmeasured | adoption and evaluation are the next step |
Unproven, explicitly: S2/S3 general detection performance (synthetic cases only), production detection and false-alarm rates, doubt outside the failure vocabulary, and every failure that leaves no evidence in the receipt.
Adoption needs three concrete things: each pipeline step emits a receipt (status, count, tail); the checker runs next to the scheduler on a schedule; and a flag is operated as "review", never as "definitely failed".
Reproduce
Every number above re-derives from the repo, and all 14 tests pass.
sunnydachs
/
silent-failure-watch
False-success evidence from a real unattended agent fleet: 1,000 cron executions + a controlled injection grid
silent-failure-watch
English | 日本語
Evidence repo for the article series on false success in unattended agent pipelines: agents that report a job as done when the artifact, the data, or the run itself never happened.
Two datasets live here, both measured on a real single-machine agent fleet (a WSL2 laptop running cron-driven agents, 1,000 tracked executions exported 2026-10-24):
-
data/fleet_runs.jsonl— sanitized export of 1,000 real cron executions (status, generic job type, error class, duration, schedule lateness) Reproduce the summary withpython3 scripts/analyze_fleet.py. -
experiments/inject/— a small controlled grid: 4 environmental failure classes injected into a toy extraction task; each run scores the agent's own structured status report against a rule-based, state-checking verifier. Reproduce withpython3 experiments/inject/run_grid.py.
The external anchor is arXiv:2606.09863 ("From Confident Closing to Silent Failure"), which measures false success across 9,876 tau2-bench and 1,879 AppWorld trajectories. This repo contributes the two things a benchmark cannot…
# tests (14 passed)
python3 -m pytest tests/ -q
# model A grid (40 receipts -> 8/8 caught, 0 false alarms)
python3 scripts/receipt_check.py --eval
# model B grid (34 receipts -> 7/7 caught, 0 false alarms)
python3 scripts/receipt_check.py --eval results/inject_results_b.jsonl
# boundary suite (14 cases -> 5/6 caught)
python3 scripts/receipt_check.py --eval tests/fixtures/receipt_cases.jsonl
The previous post — the 1,000-execution fleet and the injection grid — is here:
This is a personal OSS project, provided without warranty. Use at your own risk — issues and improvement ideas are welcome.
Top comments (0)