We run an organization made of agents. Not a demo — a company's back office. Bookkeeping,
inbox triage, drafting, review, research. 78 agents, local models, no human in the loop
between the schedule firing and the output landing in a ledger.
For 69 days we recorded every failure. Not "the run errored" — what was wrong with the
answer. 11850 records, published as a dataset.
This is what we found. It is not what we expected.
The uncomfortable part first: none of these raised an error
Every failure below returned HTTP 200. Every one had plausible length and fluent prose.
Every one was recorded by our monitoring as a successful completion.
On one day in this period our uptime metric read 97–100% for the full day, and the number
of finished deliverables was zero. Both numbers were correct.
If your dashboard says the agent ran, you have learned that the agent ran.
The distribution
Of the records that carry a stated reason, these are the types, by how often they occurred:
| Count | Type | What it looks like |
|---|---|---|
| 4522 | too_short |
too short (0 chars, minimum 150) |
| 3319 | forbidden_present |
あってはならないものが含まれる: (お客様/顧客)の声 |
| 2414 | other |
日本語が壊れている |
| 1602 | missing_required |
missing required: 判定[:：] |
| 902 | broken_language_mix |
Japanese and English mixed into nonsense (4 English function |
| 513 | chinese_in_japanese |
Chinese characters in Japanese output: 资 |
| 406 | not_japanese |
not Japanese (no kana at all) |
| 309 | answer_to_other_question |
別の仕事の答え（採点用のJSON）が返ってきている |
| 290 | unearned_claim |
claims customers it does not have (ja) |
| 63 | invented_specifics |
材料に無い具体を書いている（readme.json, MIT） |
| 36 | reasoning_leaked |
英語の思考が本文に漏れている |
| 36 | refusal |
断り文句で終わっている |
| 3 | hedge |
逃げ口上（サンプルだと断っている） |
6124 of the 14415 type occurrences — 42% — are boring.
Not hallucination. Not jailbreaks. Not safety. The model answered — in the wrong shape.
It skipped the heading the downstream parser keys on. It wrote 0 characters. It answered
in the wrong language.
The dramatic failures everyone writes evals for — refusing (36), answering a different
question (309) — are the rarest things in the dataset. Combined they are 2.4%.
We had built our review pass around the dramatic ones.
Shape failures are the ones that cost you
missing_required is the largest single type, and it is the most expensive, because it is
the one that silently breaks a pipeline rather than one output.
Our single most frequent violation, 1602 occurrences, is one missing line:
missing required: 判定[:：]
判定 means "verdict". A review agent's contract requires a line starting with that word,
because a later stage reads that line to decide whether the reviewed item advances. The
model would write a thoughtful, correct, well-structured review, and put its verdict in
prose instead of on a line with that prefix.
The review is good. The reviewer is not wrong. The stage after it cannot read the answer.
This is what makes shape failures expensive: they are invisible to a human spot-check.
Someone reading the output sees a fine review and concludes the agent works.
A related failure, which we found separately and which the dataset does not contain:
42 items were approved by the upstream stage and produced zero downstream work. That one
was not a shape mismatch — the downstream stage was generating its input from four
hard-coded sentences and never read the upstream output at all. It had been that way from
the beginning. Both stages reported success for weeks.
Different cause, same shape of blindness: every stage was green, and nothing crossed
between them. We now record, on every produced artifact, which artifact it was made from,
so an unread output is visible as a number rather than as an absence.
Language corruption is invisible if you only read English
not_japanese (406) + chinese_in_japanese (513) + broken_language_mix (902) +
reasoning_leaked (36) = 1857 occurrences, our third-largest cluster.
Some samples, verbatim from the dataset:
Get the checklist and the three watchdogs (free) → gxcafe.co.jp/harness-kit
Chinese characters in Japanese output: 资
Japanese and English mixed into nonsense (4 English function words inside Japanese)
not Japanese (no kana at all)
The last one is worth sitting with. In a model evaluation we ran while writing this — not
in the dataset, but with the same contracts we run in production — a candidate model
produced 1,680 characters of confident, well-formatted text for a Japanese review task
containing not a single kana character. It was entirely Chinese. Length: passed.
Structure: passed. A reader who does not read Japanese would see a completed deliverable.
If you operate in a non-English language, your base model's training-data language will
leak, and it will leak in a way that passes every structural check you have. We now measure
kana presence as a hard gate, and we keep the writer model and the reviewer model in
different families so a family-wide language failure cannot approve itself.
What we changed
Four things, in order of how much they mattered.
1. Declare the shape, machine-check it, before anything downstream reads it.
Minimum length, required patterns, forbidden patterns, expected language, and — for
anything with numbers — whether the number is labeled as measured, target, assumed, or
estimated. This is the tool we extracted from that work:
honto-contract. Zero dependencies,
synchronous, works on any string. It exists because we needed it, not the other way around.
2. When something fails, record why, not that.
For our first 41 days we wrote "contract not satisfied" and threw the reasons away. 2179 of
our 11850 records carry no reason, and we cannot reconstruct them — the outputs are gone. It
took us three weeks to notice, because "0 failures with a reason" and "0 failures" look
identical on a chart. Those 2179 rows are in the published dataset, labeled, because
deleting them would misrepresent what we know.
3. Do not let the writer approve its own work.
Our reviewer model shares no family with our writer model. When we briefly used the same
family for both, the reviewer approved six consecutive outputs that were in the wrong
language.
4. Measure produced output, not invocations.
We spent weeks judging "is this agent working?" by whether it had started recently. 78
agents were permanently excluded from remediation because a scheduler touched their
start-log every hour. The number was real. It measured the wrong thing.
The data
- Dataset: GXCafe/ai-agent-failure-logs — CC-BY-4.0
- Checker: honto-contract — MIT
Both are free. The dataset is one organization's records, not a benchmark: 11850 rows over 69
days from a single deployment, mostly Japanese-language tasks, mostly small local models
(7B–35B). It will not tell you what fails in your setup. It will tell you what kind of
thing fails when nobody is watching, which is the part we could not find written down
anywhere when we needed it.
We are still running. The dataset updates as we fail.
The rest of it
We keep running, so the dataset keeps growing, and the contract templates we use in
production (76 of them) are not in this repository.
If you want either of those, or you think the data is wrong, write to us:
sales@gxcafe.co.jp. We will tell you what we can and cannot do, in the same tone as
this page.
Notes on honesty
Every number here is counted from the published dataset and reproducible from it. Where we
do not know something, this article says so:
- 2179 of 11850 records have no stated reason. We started recording reasons on day 41. Recent days run about 59% with a reason. The gap is historical and we did not backfill it.
- The type counts are occurrences, not rows; one failed output can carry several types.
- We have no customers for this and are not claiming results from it. We built the checker for our own pipeline and published it because the failure taxonomy was the part we wished someone else had published first.
If you run agents unattended
Our unattended-operation checklist and three watchdog templates are free
(email-gated):
Get the checklist and the three watchdogs (free)
More of these, as we find them: GX Cafe engineering notes — 12 posts, no signup.
Top comments (4)
"If your dashboard says the agent ran, you have learned that the agent ran" belongs on a poster. We operate a fleet of scheduled agents too, and the 97%-uptime-zero-deliverables day is the single most under-reported failure mode in this whole space — the metric and the outcome are both correct and completely disconnected.
What resonates most is that 43% of your failures are shape, not hallucination. That matches what we see: the dangerous class isn't the agent going rogue, it's the agent returning fluent, plausible-length output that skips the one heading a downstream parser keys on. Our most effective fix wasn't better prompts — it was moving the validation from "did it error" to a contract check on the output itself (required sections present, length floor, language/script assertion) that runs before anything downstream consumes it, and treats a violation as a hard failure that reschedules, not a warning.
The Japanese/Chinese script-mixing bucket is fascinating and not something I'd have thought to assert on. Are you running that check with a rule-based script detector, or a lightweight classifier? Thinking about how to generalize it beyond one target language.
Rule-based, deliberately. It's a curated character-class regex of codepoints that are simplified-Chinese-only and never appear in written Japanese (们你请让这为对… plus a few traditional-only variants). We explicitly leave out all kanji shared between the two languages — early versions flagged legitimate Japanese constantly, and a checker that cries wolf gets ignored.
A second, cheaper assertion catches the inverse case: "CJK text with zero kana is not Japanese." Japanese cannot be written without kana, so its absence in a long output is a hard signal — no classifier needed.
For generalizing: I'd assert the presence of the script the target language cannot be written without (kana for Japanese, Hangul for Korean) rather than trying to detect every possible intruder script. Presence checks have a bounded false-positive story; intruder lists grow forever.
The provenance fix is the right instinct, but "which artifact it was made from" can still be gamed by the exact failure it exists to catch - a consumer that records the source id without reading the content. Your start-log lesson is the tell: the scheduler touched the file every hour, and the number was real and meaningless. A provenance edge is the same kind of number if the consumer writes it from metadata it never had to read.
The version that survives: make the consumer cite what it actually read. Instead of recording input_artifact_id alone, record a fingerprint of the content the consumer consumed - a hash, or better, the re-derived contract fields (the 判定 line, the row count). Then "unread" stops being an absence and becomes a mismatch: the cited value does not match the artifact, and a mismatch shows up as a number instead of as nothing. The two-stage green case you describe - downstream generating its input from four hard-coded sentences - fails this check on the first run, because the cited verdict line cannot be re-derived from an artifact that was never read.
This is also the same shape as your "record why, not that" fix. A reason you did not record and a hash you did not compute are both absences that look identical to "nothing happened" on a chart. Computing the citation at consumption time is the write-side version of that discipline: you can never backfill a read receipt, so it has to be written when the reading happens.
You're right, and this comment turned into a commit. As of today:
When our reviewer receives an excerpt, we write a read receipt at hand-off time: sha256 of the exact bytes passed, excerpt length, full length. Written when the reading happens — as you said, you can never backfill a read receipt.
Queue entries now carry the sha256 + length of the deliverable content actually read at enqueue time. If the file is unreadable or empty, the item never enters the approval queue (fail-closed, archived with a reason) — that was exactly the "records the source id without reading the content" hole, and we had it.
The audit recomputes hashes against the current files, so "never read" and "swapped after enqueue" both surface as a mismatch count instead of as nothing.
The re-derived-contract-fields version (cite the 判定 line itself) is the next step; hashes were shippable today. Thanks — this was the most useful code review we've had, and it came from a blog comment.