Update — v0.3.0 released. CauterRule is now live on GitHub and PyPI. It turns repeated agent failures into permanent standing rules — extract, replay-test, promote.
pip install cauterulegives you the full CLI, framework adapters, rule lifecycle, pack ecosystem, and official rule packs. The v0.3.0 field test report evaluated 2 cloud models across 40 corpora and 4,768 trajectory-runs and is the source for every number below. Release notes · Changelog
CauterRule is an open-source sidecar that learns standing rules from repeated agent failures. It extracts lessons from trajectories, replay-tests them, and tries to separate reusable guidance from noisy overgeneralization.
Here is the failure mode that never makes it into a paper. You launch a 4,768-run benchmark. It churns for an hour. Then one request to the model provider simply never returns — no exception, no error, no timeout. The worker is blocked. The queue drains to zero throughput. You have not lost one trajectory. You have lost the sweep, and the API spend it already consumed.
v0.3.0 is the first CauterRule field test that finished. The reason is not a better prompt. It is five small guards in the runner.
The assumptions we started with (all of them wrong at scale)
Our original runner encoded three assumptions that hold at 50 runs and fail at 4,768:
Assumption 1: the provider will always respond.
Assumption 2: a timeout is a transient error, so retry it.
Assumption 3: a failed trajectory should be retried until it succeeds.
At scale, assumption 1 produces infinite hangs, assumption 2 doubles the cost of a request that was already dead, and assumption 3 turns one bad input into an unbounded retry loop. None of these are model problems. They are harness problems — and they compound.
The scale that breaks them: 40 corpora × 2 models = 4,768 runs, roughly 2,384 trajectories per model, several LLM calls per trajectory.
The fix: five guards (#713)
None of this is clever. It is the set of things you must do before you can trust a large sweep:
| Guard | Implementation | Failure it removes |
|---|---|---|
| Per-trajectory timeout |
future.result(timeout=120) (client create(timeout=30)) |
A hung call blocks a worker forever |
| Non-retryable timeouts | classify timeout as terminal | Retrying a dead request multiplies cost |
| Token cap | max_tokens=4096 |
Runaway generations blow latency and spend |
| Quarantine | CAUTERULE_QUARANTINE_IDS |
Known-bad IDs are skipped, not retried |
| Cancel on shutdown | executor.shutdown(wait=False, cancel_futures=True) |
A stuck thread blocks ThreadPoolExecutor.__exit__
|
The shape of the concurrency fix — note it is the per-future result(timeout), not a with block, that does the work. The with ThreadPoolExecutor context manager's __exit__ calls shutdown(wait=True), which blocks until every thread finishes, so a single stuck thread would hang the whole sweep:
futures = {executor.submit(run_trajectory, t): t for t in trajectories}
for fut in as_completed(futures):
try:
collect(fut.result(timeout=120)) # per-trajectory watchdog
except TimeoutError:
mark(futures[fut], "timeout") # record it, keep sweeping
executor.shutdown(wait=False, cancel_futures=True) # never wait on a stuck thread
The principle: one bad request costs one trajectory, not the sweep.
The data
What the hardening bought, concretely:
| Metric | Value |
|---|---|
| Trajectory-runs completed | 4,768 |
| Models | 2 (gpt-4o-mini, llama-3.1-8b) |
| Corpora | 40 |
| Safety gate dropped (per model) | 580 trajectories |
| LLM calls avoided by the gate (per model) | 1,160 |
| Safety silence (successes / negatives) | 60/60 · 60/60 |
| Adversarial promotions | 0 |
The safety number is only credible because every corpus finished. A benchmark that completes its easy corpora and dies on the hard ones is measuring the corpora it could finish.
Bonus: cost became a first-class metric
Once runs stop dying, you can afford to measure spend. v0.3.0 captures token usage on every call and prices it with real provider rates:
@dataclass
class LLMResponse:
text: str
prompt_tokens: int = 0
completion_tokens: int = 0
| Model | Input ($/1M) | Output ($/1M) |
|---|---|---|
| gpt-4o-mini | 0.15 | 0.60 |
| llama-3.1-8b | 0.06 | 0.06 |
The SDK was already returning usage on every response. We were discarding it. Adding two fields turns "what does 1,000 trajectories cost?" from a shrug into a measured answer — the question that actually gates adoption.
What worked
- The sweep completed end to end. 40 corpora, 2 models, 4,768 runs.
-
The motivating hang is now a non-event. The original
ci-fail-015prompt drove a local model into infinite repetitive generation and blocked the sweep indefinitely; with the per-future watchdog it completes in ~3.7s andraw/ciruns 110/110. - Fail-soft held. Timeouts and quarantine dropped broken runs, not hard cases — the results stayed comparable.
- Cost transparency fell out for free. A few lines of token accounting, using data already on the wire.
- A single hung call is now a rounding error. The failure mode that motivated the work is gone.
What didn't work
-
Local models did not survive contact with scale. The same hardening made it undeniable that the local OMLX setup was too slow and hung on
raw/ci. The field test went cloud-only. The "ship on local models" path from the v0.1.0 report did not hold at 40 corpora. -
We built measurement tools and didn't run the protocols. Cross-session repeat-failure reduction and human-vs-replay agreement both have scripts (
cross_session.py,human_agreement.py) and runner flags. Neither protocol was executed. A flag that was never pulled is not evidence. - We didn't instrument the drops. The runner discards timed-out trajectories cleanly, but we did not log the timeout distribution. The shape of the failure — where and how often calls hang — is itself data we threw away.
Questions we still can't answer
- Is 120s the right timeout, or does it truncate legitimately slow multi-step extractions? We never measured how close normal calls come to the ceiling.
- Do the timeouts cluster in
raw/ci, or are they spread evenly? We can't say without per-drop logging. - Does quarantine mask a provider regression? Skipping known-bad IDs is safe for a one-off sweep; in a continuous loop it could hide a systematic failure.
- What is the actual $/1k-trajectory number per corpus, not just per model? We have per-model pricing but haven't published per-corpus costs.
What I learned
Throughput is a product feature for evaluation. Your benchmark's quality ceiling is whether it finishes. A harness that can't survive a hung call can't run the corpora where the interesting failures live.
Fail-soft beats fail-hard. Retries that assume transience are a cost bug waiting to happen. Classify failures as terminal vs. transient explicitly.
Capture usage while you're already holding the client. Token accounting is trivial when you own the call site and impossible when you don't. Do it from day one.
Building the instrument is not taking the reading. We shipped cross-session and agreement tooling and left both protocols unrun, so those questions are still open. Tooling is not measurement.
The broader lesson
If you run large evaluations against a flaky external API, the infrastructure is not a distraction from the science — it is part of the science.
The most valuable code in CauterRule's v0.3.0 field test was not a matcher or a prompt. It was five guards that let 4,768 requests finish, plus two dataclass fields that let us price them. That is what turned a benchmark that kept dying into one you can trust — and, conveniently, one that can tell you what it costs.
References
- CauterRule v0.3.0 release notes
- v0.3.0 field test report
- Cost measurement (token capture + per-model pricing)
- Field test plan (corpora, models, runner flags)
- Generated results (all 40 corpora)
- PRD — architecture
- User guide · Changelog
CauterRule v0.3.0 is released. The methodology, hardening notes, and cost report are in the field test report and the cost measurement doc. The repo is public. Install with
pip install cauterule. Changelog · Release notes
Top comments (0)