DEV Community: GX Cafe LLC

Your monitoring says healthy. Your agents are not.

GX Cafe LLC — Thu, 03 Sep 2026 15:04:17 +0000

What Broke

In the weeks leading up to our most recent production run, our local LLM agent fleet experienced a series of failures that went unnoticed for days. Looking at the failure ledger, several patterns emerge:

Process Errors in Inference: Multiple entries show process_error failures during the secretary/consult-classify and secretary/consult-fields stages. For example:
- On August 29th at 08:06:11 UTC, the ollama/qwen3.8:27b model encountered a process_error with a duration of 318,454ms.
- On August 30th at 09:54:39 UTC, the same model failed again with an even longer duration of 1,813,103ms.
Timeouts in Agent Evaluation: On September 1st, two consecutive timeouts occurred during hr/agent-evaluation:
- 02:55:03 UTC and 03:00:06 UTC, both with the ollama/qwen3.8:27b and qwen3.6:27b models, respectively. Each timeout lasted approximately 300 seconds.

These failures indicate that while the agents were running unattended, they encountered issues that prevented them from completing their tasks. The long durations and timeouts suggest that the models were either stuck in a loop or encountered resource constraints that were not being addressed.

Why It Happened

The root cause of these failures can be attributed to several factors:

Model Instability: The models being used, such as ollama/qwen3.8:27b and ollama/qwen3.6:27b, may have inherent instability when running continuously without human oversight. This could be due to memory leaks, inefficient garbage collection, or other performance issues.
Resource Constraints: Running multiple agents simultaneously can strain system resources, leading to timeouts and process errors. The long durations recorded in the failure ledger suggest that the agents were consuming excessive CPU or memory, causing the system to become unresponsive.

Get the checklist and the three watchdogs (free) → gxcafe.co.jp/harness-kit

Lack of Monitoring: Without proper monitoring in place, these failures went unnoticed for an extended period. This highlights the importance of having real-time alerts and monitoring tools to detect and address issues promptly.

What We Put in Place

To prevent similar issues in the future, we implemented several mechanical guards:

Automated Alerts: We set up automated alerts to notify the engineering team immediately when a process_error or timeout occurs. This ensures that any issues are addressed promptly, minimizing downtime and preventing prolonged failures.
Resource Monitoring: We implemented resource monitoring to track CPU and memory usage of the agents. If an agent exceeds predefined thresholds, it is automatically restarted or reconfigured to use fewer resources.
Model Health Checks: Regular health checks are now performed on the models to ensure they are running optimally. This includes checking for memory leaks, optimizing garbage collection, and ensuring that the models are not consuming excessive resources.
Failover Mechanisms: We introduced failover mechanisms to switch to a backup model or agent if the primary one fails. This ensures that the agent fleet can continue operating even if individual components encounter issues.

The Lesson

The key lesson from this experience is the importance of proactive monitoring and maintenance in unattended agent fleets. While running agents unattended can increase efficiency and productivity, it also introduces risks that must be managed carefully. By implementing automated alerts, resource monitoring, regular health checks, and failover mechanisms, we can ensure that our agent fleet remains stable and reliable, even when running for extended periods.

In conclusion, the 58-day unattended run highlighted the silent failures that can occur in local LLM agent fleets. By addressing these issues with the mechanical guards we put in place, we have improved the resilience and reliability of our system. Moving forward, we will continue to refine our monitoring and maintenance strategies to ensure that our agent fleet operates smoothly and efficiently.

Get the checklist and the three watchdogs (free) → gxcafe.co.jp/harness-kit

You can never backfill a read receipt: a blog comment fixed our AI pipeline

GX Cafe LLC — Mon, 31 Aug 2026 23:23:55 +0000

Last week we published a post-mortem: our AI reviewer hallucinated a request
that didn't exist, and our producer retried the same document 245 times. A
reader, pm25coder, left a comment that turned out to be the best code review
we've ever received. This post is what we shipped because of it, same day.

The comment, compressed

Our fix at the time was provenance: record which artifact each output was
made from. The comment pointed at the hole:

"Which artifact it was made from" can still be gamed by the exact failure
it exists to catch — a consumer that records the source id without reading
the content. [...] Make the consumer cite what it actually read. [...]
You can never backfill a read receipt, so it has to be written when the
reading happens.

He was right, and we checked: our approval queue recorded the path of
every deliverable. Nothing anywhere proved the enqueuer had ever opened the
file. An empty or deleted file could sit in the human approval queue
looking exactly like a real one.

What we shipped

1. Read receipts at hand-off. When the reviewer receives an excerpt, we
now write {sha256(excerpt), excerptChars, fullChars} at the moment of
hand-off. Written at read time, because it cannot be written later.

2. Fingerprints at enqueue. A deliverable enters the approval queue only
after the enqueuer actually reads it: content hash and length are stored on
the queue entry. Unreadable or empty → it never enters the queue, and the
reason is archived where a human can see it.

3. Re-derivable citations. The queue entry also quotes the reviewer's
verdict line verbatim. The nightly audit re-extracts that line from the
review file and compares. A citation that cannot be re-derived from the
artifact means someone wrote it without reading — and it now shows up as a
mismatch count, not as an absence. That was the commenter's core point:
an unread input should be a number on a chart, not a blank space.

The check had the bug it hunts

Get the checklist and the three watchdogs (free) → gxcafe.co.jp/harness-kit

Worth confessing: our first version skipped any queue entry that lacked a
hash — if (!entry.sha256) continue; — which meant a forged citation
without a hash sailed through the very audit built to catch forged
citations. We only found it because we test every check by faking the
failure it should catch, and the fake passed. If you build watchdogs:
break them on purpose before you trust them. Ours needed it, again.

Numbers, for honesty

2,038 reviews audited for invented terms: 4 contaminated (0.2%)
Those 4 caused ~470 wasted generations before anything gave up
Forged-citation probe after the fix: caught as a mismatch, first try

If you run agents unattended

The output-contract checker from this story is free on npm:
honto-contract.

Our unattended-operation checklist and three watchdog templates are free
(email-gated):

Get the checklist and the three watchdogs (free)

More of these, as we find them: GX Cafe engineering notes — 12 posts, no signup.

The full production set — now including the payment gate we extracted this
week — is on the same page.

Honest note: still no customers. Everything above is what we run on
ourselves, measured on our own failures. And thanks again, pm25coder —
comments like yours are why we write these.

Your local LLM app needs guardrails before it needs prompts

GX Cafe LLC — Thu, 27 Aug 2026 14:03:17 +0000

Most local-LLM tutorials start with the fun part: the prompt. After running a fleet
of autonomous agents on local models 24/7 and logging every failure — the ledger now
holds over eight thousand entries — we start somewhere else.

Here is the single most important thing that ledger taught us: the call succeeding
tells you nothing. The majority of our contract violations were outputs that were
too short, or empty — while the API returned success. The model answered. The
dashboard was green. Nothing was produced.

So we distilled the survival kit into a scaffold you can stand up in one command:

npx create-local-llm-app my-app

Five files, about 180 lines, plain Node and Express, no framework lock-in. What gets
wired in:

1. An output contract

A contract declares what "done" looks like — and then verifies the artifact itself.
Never a proxy. Not the exit code, not a log line, not HTTP 200. contract.js has
three kinds of clause:

minChars — the cheapest check, and statistically the most valuable
must — patterns that have to appear
mustNot — leftover placeholder tags, unfinished markers, refusal leakage ("as an AI, I cannot…" inside what was supposed to be a business document)

Every mustNot entry exists because it caught a real production failure.

2. Retry that feeds failures back

Get the checklist and the three watchdogs (free) → gxcafe.co.jp/harness-kit

When the contract rejects an output, the failure reason goes into the next prompt.
Three strikes and the item is rejected outright — there is no "accept with warnings",
because accept-with-warnings is "apologize later" with extra steps.

3. An approval queue

Nothing the model writes leaves the app without a person pressing approve. No
decision means no. That rule comes from experience: an unattended script once
contacted a real company because its default was "send". The dangerous side is
never the default here.

4. Two watchdogs

A heartbeat (is the process alive?) and a silent-zero check (is anything actually
being produced?). These are different questions. Running is not the same as
producing — a fleet can be 100% "up" with zero output all day, and nothing in a
standard health check will tell you.

Why so small?

Because the guardrails are the product; the model is replaceable. Swap Ollama
models in .env, replace the prompts with your own business task, and the
contract/queue/watchdog skeleton stays exactly where it is.

If you want to go deeper, the checker we run in production is free and MIT:
honto-contract. A free snapshot of
the failure ledger behind all of this is on
Hugging Face.

Start with the scaffold, break every guard on purpose, and watch them hold. That is
the fastest way to trust your own unattended setup.

If you run agents unattended

Our unattended-operation checklist and three watchdog templates are free
(email-gated):

Get the checklist and the three watchdogs (free)

More of these, as we find them: GX Cafe engineering notes — 12 posts, no signup.

Our AI reviewer invented a request. Our producer retried 245 times.

GX Cafe LLC — Mon, 24 Aug 2026 00:31:03 +0000

We run ~100 LLM agents unattended on local models. Last week we found one
document that had been rewritten 245 times in 5 days — every attempt
rejected. A sibling document: 225 times. Combined, about 470 wasted
generations, all burned on the same two files.

Here is the autopsy, with the actual numbers.

The loop

Our pipeline is simple: a producer agent writes a document, a reviewer agent
checks it against a contract (minimum length, required sections, no
placeholder junk), and rejected work goes back with fix instructions.

The rejected document was a key-management (KMS) implementation spec —
4,452 characters, perfectly on-topic. The reviewer's verdict:

"The request was a 3-line email triage response (LOCK / VERDICT / REASON),
but the answer is a long KMS spec. Rewrite as 3 lines only."

One problem. We grepped the document: the words "LOCK", "VERDICT", and the
name of the triage service appear zero times in it. The reviewer had
invented the request.

Why the loop never ended

Two contracts collided:

The reviewer's fix instruction: output 3 lines only
The producer's output contract: minimum 600 characters

No output can satisfy both. So the producer failed the contract, got
re-queued, produced again, failed again — 245 times. Our retry cap counted
reviews, but a contract-failed output never reaches review. The give-up
mechanism existed; it just watched the wrong counter.

Root cause: the reviewer never saw the request

Our review prompt contained the artifact body (first 4,000 chars) and the
output format. It never contained the original request. We asked a model
"does this match the request?" without telling it what the request was.
A model asked to judge against information it doesn't have will
hallucinate that information. Ours did, confidently, 245 times' worth.

Get the checklist and the three watchdogs (free) → gxcafe.co.jp/harness-kit

Bonus failure: we truncated long documents to 4,000 characters before
review without saying so, and reviewers marked them "thin — cut off
mid-sentence." The cut was ours, not the producer's.

How common was it?

We audited all 2,038 reviews on file for concrete terms (product names,
format tokens) that appear in the review but nowhere in the reviewed
document. Result: 4 contaminated reviews — 0.2%.

That's the uncomfortable lesson: a 0.2% hallucination rate produced 470
wasted runs, because nothing ever gave up. Low rate × infinite retries =
unbounded damage. The rate is not the risk; the loop is.

The fixes (all mechanical)

Pass the original request into the review prompt. If it can't be extracted, the prompt now says: "do NOT guess the request — say it is unknown and judge the artifact on its own."
Declare truncation. "First 4,000 of 8,784 chars — the cut is ours."
Reject impossible instructions at the review's own exit gate. A review demanding "N lines only" while the production contract requires 600+ chars now fails as a review and never enters the queue.
Count consecutive contract failures, not just reviews, and park the item for a human after 5 — with the last verdict and fix instruction attached, so the human can see why in one glance.

Each fix ships with a test we deliberately broke to confirm it fails.

If you run agents unattended

The checker that catches broken outputs in this story (empty text, language
leakage, placeholder junk, contract violations) is free on npm:
honto-contract — it passed
600 downloads last week, so somebody besides us finds this useful now.

The unattended-operation checklist and three of our watchdog templates are
free (email-gated):

Get the checklist and the three watchdogs (free)

More of these, as we find them: GX Cafe engineering notes — 12 posts, no signup.

The full production set (7 templates — cron registry, silent-zero watch, heartbeat, output contracts: the exact ones in this story) is available on the same page.
(https://gxcafe.co.jp/harness-kit/?utm_source=devto&utm_medium=article&utm_campaign=harness-kit&utm_content=20260824-kiji-c9185d).

Honest note: we have no customers yet. Everything above is exactly what we
run on ourselves, measured on our own failures.

What actually breaks when you run LLM agents unattended for 58 days

GX Cafe LLC — Fri, 21 Aug 2026 11:26:01 +0000

We run an organization made of agents. Not a demo — a company's back office. Bookkeeping,
inbox triage, drafting, review, research. 78 agents, local models, no human in the loop
between the schedule firing and the output landing in a ledger.

For 69 days we recorded every failure. Not "the run errored" — what was wrong with the
answer. 11850 records, published as a dataset.

This is what we found. It is not what we expected.

The uncomfortable part first: none of these raised an error

Every failure below returned HTTP 200. Every one had plausible length and fluent prose.
Every one was recorded by our monitoring as a successful completion.

On one day in this period our uptime metric read 97–100% for the full day, and the number
of finished deliverables was zero. Both numbers were correct.

If your dashboard says the agent ran, you have learned that the agent ran.

The distribution

Of the records that carry a stated reason, these are the types, by how often they occurred:

Count	Type	What it looks like
4522	`too_short`	`too short (0 chars, minimum 150)`
3319	`forbidden_present`	`あってはならないものが含まれる: (お客様/顧客)の声`
2414	`other`	`日本語が壊れている`
1602	`missing_required`	`missing required: 判定[:：]`
902	`broken_language_mix`	`Japanese and English mixed into nonsense (4 English function`
513	`chinese_in_japanese`	`Chinese characters in Japanese output: 资`
406	`not_japanese`	`not Japanese (no kana at all)`
309	`answer_to_other_question`	`別の仕事の答え（採点用のJSON）が返ってきている`
290	`unearned_claim`	`claims customers it does not have (ja)`
63	`invented_specifics`	`材料に無い具体を書いている（readme.json, MIT）`
36	`reasoning_leaked`	`英語の思考が本文に漏れている`
36	`refusal`	`断り文句で終わっている`
3	`hedge`	`逃げ口上（サンプルだと断っている）`

6124 of the 14415 type occurrences — 42% — are boring.

Not hallucination. Not jailbreaks. Not safety. The model answered — in the wrong shape.
It skipped the heading the downstream parser keys on. It wrote 0 characters. It answered
in the wrong language.

The dramatic failures everyone writes evals for — refusing (36), answering a different
question (309) — are the rarest things in the dataset. Combined they are 2.4%.

We had built our review pass around the dramatic ones.

Shape failures are the ones that cost you

missing_required is the largest single type, and it is the most expensive, because it is
the one that silently breaks a pipeline rather than one output.

Our single most frequent violation, 1602 occurrences, is one missing line:

missing required: 判定[:：]

判定 means "verdict". A review agent's contract requires a line starting with that word,
because a later stage reads that line to decide whether the reviewed item advances. The
model would write a thoughtful, correct, well-structured review, and put its verdict in
prose instead of on a line with that prefix.

The review is good. The reviewer is not wrong. The stage after it cannot read the answer.

This is what makes shape failures expensive: they are invisible to a human spot-check.
Someone reading the output sees a fine review and concludes the agent works.

A related failure, which we found separately and which the dataset does not contain:
42 items were approved by the upstream stage and produced zero downstream work. That one
was not a shape mismatch — the downstream stage was generating its input from four
hard-coded sentences and never read the upstream output at all. It had been that way from
the beginning. Both stages reported success for weeks.

Different cause, same shape of blindness: every stage was green, and nothing crossed
between them. We now record, on every produced artifact, which artifact it was made from,
so an unread output is visible as a number rather than as an absence.

Language corruption is invisible if you only read English

not_japanese (406) + chinese_in_japanese (513) + broken_language_mix (902) +
reasoning_leaked (36) = 1857 occurrences, our third-largest cluster.

Some samples, verbatim from the dataset:

Get the checklist and the three watchdogs (free) → gxcafe.co.jp/harness-kit

Chinese characters in Japanese output: 资
Japanese and English mixed into nonsense (4 English function words inside Japanese)
not Japanese (no kana at all)

The last one is worth sitting with. In a model evaluation we ran while writing this — not
in the dataset, but with the same contracts we run in production — a candidate model
produced 1,680 characters of confident, well-formatted text for a Japanese review task
containing not a single kana character. It was entirely Chinese. Length: passed.
Structure: passed. A reader who does not read Japanese would see a completed deliverable.

If you operate in a non-English language, your base model's training-data language will
leak, and it will leak in a way that passes every structural check you have. We now measure
kana presence as a hard gate, and we keep the writer model and the reviewer model in
different families so a family-wide language failure cannot approve itself.

What we changed

Four things, in order of how much they mattered.

1. Declare the shape, machine-check it, before anything downstream reads it.
Minimum length, required patterns, forbidden patterns, expected language, and — for
anything with numbers — whether the number is labeled as measured, target, assumed, or
estimated. This is the tool we extracted from that work:
honto-contract. Zero dependencies,
synchronous, works on any string. It exists because we needed it, not the other way around.

2. When something fails, record why, not that.
For our first 41 days we wrote "contract not satisfied" and threw the reasons away. 2179 of
our 11850 records carry no reason, and we cannot reconstruct them — the outputs are gone. It
took us three weeks to notice, because "0 failures with a reason" and "0 failures" look
identical on a chart. Those 2179 rows are in the published dataset, labeled, because
deleting them would misrepresent what we know.

3. Do not let the writer approve its own work.
Our reviewer model shares no family with our writer model. When we briefly used the same
family for both, the reviewer approved six consecutive outputs that were in the wrong
language.

4. Measure produced output, not invocations.
We spent weeks judging "is this agent working?" by whether it had started recently. 78
agents were permanently excluded from remediation because a scheduler touched their
start-log every hour. The number was real. It measured the wrong thing.

The data

Dataset: GXCafe/ai-agent-failure-logs — CC-BY-4.0
Checker: honto-contract — MIT

Both are free. The dataset is one organization's records, not a benchmark: 11850 rows over 69
days from a single deployment, mostly Japanese-language tasks, mostly small local models
(7B–35B). It will not tell you what fails in your setup. It will tell you what kind of
thing fails when nobody is watching, which is the part we could not find written down
anywhere when we needed it.

We are still running. The dataset updates as we fail.

The rest of it

We keep running, so the dataset keeps growing, and the contract templates we use in
production (76 of them) are not in this repository.

If you want either of those, or you think the data is wrong, write to us:
sales@gxcafe.co.jp. We will tell you what we can and cannot do, in the same tone as
this page.

Notes on honesty

Every number here is counted from the published dataset and reproducible from it. Where we
do not know something, this article says so:

2179 of 11850 records have no stated reason. We started recording reasons on day 41. Recent days run about 59% with a reason. The gap is historical and we did not backfill it.
The type counts are occurrences, not rows; one failed output can carry several types.
We have no customers for this and are not claiming results from it. We built the checker for our own pipeline and published it because the failure taxonomy was the part we wished someone else had published first.

If you run agents unattended

Our unattended-operation checklist and three watchdog templates are free
(email-gated):

Get the checklist and the three watchdogs (free)

More of these, as we find them: GX Cafe engineering notes — 12 posts, no signup.