The Watchdog That Would Have Passed Every Test and Hung Anyway

TL;DR: A watchdog fixes a stall only when timeout errors are terminal and non-retryable; idle and absolute budgets cover different waits. That is the shape to hunt for after reading the Field Notes slice log.

You have a timeout. The test turns red when the provider goes silent. The build turns green when you add the watchdog. Then a real stalled turn sits there, your session stays busy, and somebody has to intervene by hand. A retryable timeout would have let the watchdog fire, let the criteria pass, and restarted the exact stall that caused the timeout.

In this note

• The timeout that fixes nothing
• Why one timeout cannot do two jobs
• What to inspect in your own pipeline
• The second hang hiding behind the first
• Your next failure drill

That was the defect behind SLICE-012 in the Ranex harness fork. A stalled provider stream had no timeout, so its coordinator never settled and the active session stayed busy. The shipped change makes that stream reach a terminal state on its own. Good. The more useful part is the hole the tests could have missed.

The session would still hang anyway.

The timeout that fixes nothing

A timeout is only a fix when its failure path reaches a terminal state. If your retry policy recreates the condition that timed out, retrying it is a loop, not recovery.

Picture the order of events. A provider stream stops producing. The watchdog fires. The runner sees an error it is allowed to retry. It starts the stream again. The provider stalls again. The watchdog fires again. Your test can prove the timer fired. It can prove the retry happened. It can still miss the fact that the turn never ends.

SLICE-012 made the classification explicit: both watchdog failures are typed, non-retryable errors. That detail is load-bearing. The earlier prototype had avoided the retryability problem by accident. A pre-implementation review found that no requirement had said the timeout must be non-retryable, so the completed slice made the decision and required a test that proves it is not retried.

That is a small wording change with a large operational consequence. “Times out” is not enough. “Fails terminally, without retry” says what the system is allowed to do next.

If retry recreates the failure condition, retry is a hang with better logging.

If retry would only restart the stall, the failure class needs to say so. Your operator needs a terminal state, not a more energetic version of stuck.

Why one timeout cannot do two jobs

You need two thresholds when the waiting patterns differ by orders of magnitude. An idle deadline catches silence between chunks; an absolute budget limits the whole turn.

The slice wraps the stream with two separate controls. The idle deadline resets every time a chunk arrives. It answers: “Did the provider go quiet in the middle of a response?” The absolute budget races against the whole consumer. It answers: “Has this turn taken too long, even though activity continues?” Each one has to be proven without the other.

That independence matters. Feed chunks often enough and idle will never fire; only the absolute budget can stop an overlong turn. Disable idle and stall the stream; absolute must still cut it. One silent-stream fixture cannot prove both. It only proves whichever control fires first.

The first pull is deliberately untimed by idle. The records explain why: inter-chunk gaps run around 10–100ms, while a reasoning model can take minutes to its first token. One number cannot serve both distributions. An idle threshold tight enough for the gaps would cut the first response on every call.

There is a cost to that choice, and it is written down instead of tucked behind a pleasant name. A provider that accepts the connection and never sends a chunk is bounded only by the absolute budget. At the default, that can be up to 30 minutes. The record does not claim that setting is ideal. It names the limitation: a separate first-chunk budget, identified as the proper fix and left out of this slice.

Do not let one comforting timeout setting pretend it covers three different waits. First response, inter-chunk silence, and whole-turn duration need their own evidence. If you cannot show that evidence, say what remains unbounded.

What to inspect in your own pipeline

You can find this class of failure without adopting Ranex. Start where a streaming call, a retry policy, and a session state meet. The job is to make the broken path fail before you trust the repaired one.

• Find every timeout and write down its failure class: retryable, non-retryable, interrupt, or something else.
• For each retry, ask whether the next attempt changes the condition that failed. If it does not, prove the system settles instead of cycling.
• Separate time to first token, gaps between chunks, and total work time. Do not reuse a threshold merely because it is nearby.
• Make a stream send one chunk and then stall. Confirm the session reaches a terminal state without a person stopping it.
• Make a healthy slow stream complete below the idle threshold and below the whole-turn budget. A watchdog that cuts legitimate work is not a watchdog you can trust.
• Run the timeout while tool work is in flight. Check the actual tool state and settlement, not a log line that says cleanup happened.
• Use a non-default timeout and prove behavior changes. Reading a configuration value back is not proof that it controls the running system.

There is no glamorous trick here. You are looking for the exit from failure, not the detection of failure. That distinction saves you from a green test whose only achievement is proving a timer owns a clock.

The second hang hiding behind the first

A watchdog failure is an error, not an interrupt. That difference left tool fibers dispatched during streaming uncleared, and the later settlement wait held the turn open forever.

This is the defect that showed up while building the fix. Cleanup had been tied to interrupts. The watchdog produced a typed error instead, so the cleanup did not run. The runner then waited for tool fibers that had not been cleared. The visible timeout existed; the terminal state did not.

The completed slice required a specific observable for this path: the tool fiber terminates and the tool is recorded as interrupted. That is stronger than a feeling that a tool was not stranded. It makes the question checkable after the fact.

That is also why a green test suite deserves a hard question. What did it actually exercise? In the slice record, the unsafe baseline had to hang a real provider stream inside the runner. Testing a timeout helper in isolation would have been decoration. The runner, the failure class, the retry behavior, the tool cleanup, and the terminal session state all had to meet in the same proof.

SLICE-012 closed on 2026-08-07 with all nine criteria met, landing as commit 23d6a5b4ee in anthonykewl20/ranex-harness. This is harness work, not a claim that Ranex is ready for use. Ranex is pre-release; much of the broader system is designed, not built. The watchdog is one shipped durability claim in the harness fork.

If you want the surrounding model for why the verdict must sit outside an agent’s own loop, read how the kernel works. The slice record itself is in the Ranex repository under docs/slices/done/.

Questions people actually ask

These answers explain timeout failures, terminal states, and the waits each budget covers.

Why can a watchdog timeout still leave a session hanging?

A retryable timeout restarts the same stall when the condition that timed out will recur after retry. The watchdog can fire while the turn still never reaches a terminal state.

Why use both idle and absolute timeouts?

An idle deadline detects a quiet mid-stream provider, while an absolute budget bounds the whole turn even when chunks keep arriving.

Does an idle timeout cover time to first token?

No. The first pull is deliberately untimed because inter-chunk gaps and reasoning-model time to first token have very different latency distributions. A connection that never sends is bounded by the absolute budget.

What happened to tool work when the watchdog failed?

The watchdog produced an error rather than an interrupt, so streaming tool fibers were not cleared and settlement could keep the turn open. The slice added coverage for that path.

Your next failure drill

Take one timeout in your pipeline this week. Do not start by making the timer shorter. Stall the real operation after it begins. Then follow the failure all the way through: retry decision, cleanup, settlement, and the state your operator sees.

Write the failure class down. Split thresholds that are serving different distributions. Name what remains unbounded.

Try it. Break it. Tell me what broke. If this record helps you find a test that passes while its failure loops forever, give the repository a GitHub star and send an honest critique. The critique is more useful than applause.

Disclosure: this post was drafted with AI assistance. Every factual claim traces to the repository’s README or slice records — the same fact gate the product enforces on code. It ships only after Anthony’s own review.