The experiment was fine. The runner was the bug.
A single hung API call discarded hours of completed work, and I kept blaming the data.
If your long-running eval keeps dying at 80%, this is the five-part fix I wish I'd written first.
I run field tests for CauterRule, an OSS sidecar that learns standing rules from repeated agent failures. I had spent weeks on extraction, matching, and scoring — and almost no time on the process that runs everything.
A Partial Sweep Still Looks Like Data
The first time the benchmark died, I assumed the corpus was the problem. It died again. Same place.
One hung API call had discarded 60 completed audits. No partial report, no error record, just a run that stopped and a directory that looked emptier than it should.
That's the trap: a partially completed sweep doesn't announce itself. It produces a file of results that looks like data. If you report on it, you've quietly changed your denominator, and you won't know which numbers are real.
A hang is not a performance issue. It's a data-integrity issue.
The Five-Part Fix
| Failure mode | Before | After |
|---|---|---|
| Hung API call | run dies at 80%, 60 audits lost | per-trajectory timeout + non-retryable classification + quarantine |
| Runaway response | uncapped tokens | token cap per trajectory |
| Crash mid-sweep | partial report, no record | cancel-on-shutdown + quarantine record |
| Cost tracking | not captured | token capture + per-model pricing |
The mechanisms, in plain terms:
- Per-trajectory timeout — no single call can hang the sweep.
-
Non-retryable timeout classification — a timeout is recorded as
timeout, not retried forever or mistaken for a model failure. - Token cap — a runaway response can't silently inflate a run.
- Quarantine on failure — a failed trajectory is set aside with its error, not dropped.
- Cancel-on-shutdown — an interrupted sweep writes what it has and marks the rest, instead of vanishing.
Tracked in issue #713. The result was a clean field test of 4,768 trajectory-runs across 40 corpora and 2 cloud models, with zero lost sweeps — see the v0.3.0 field test report. Per-model cost capture is in the cost measurement doc.
Why This Is the Unglamorous Part
Resilience work feels like a distraction from the "real" ML problem. But every long run that dies halfway is silently poisoning the results, because a partially completed sweep still looks like data — and you won't know which run was real.
The fix wasn't a better model or a bigger machine. It was treating a single hung call as a first-class failure state instead of an accident to re-run.
Lessons
- A hang is a data-integrity bug, not a performance bug.
- Fail closed and record it. A quarantined failure is worth more than a silent gap.
- Cap the blast radius. Per-trajectory limits beat per-sweep heroics.
- Track cost per model from the start — you can't reason about routing without it.
The Honest Limitation
Runner hardening adds latency and complexity, and a timeout can mask a real model problem by treating it as infrastructure noise. I chose to fail closed and record every timeout, but the boundary between "transient" and "broken" is judgment, not science.
Not every hang is infrastructure. Some are the model genuinely struggling — and hardening the runner means you must still read the quarantine to tell the difference.
What's the ugliest piece of infrastructure your eval depends on? Mine was a runner I didn't respect for three weeks.
Code and receipts: CauterRule · v0.3.0 field test report · cost measurement · CHANGELOG — all MIT, all public.
Top comments (0)