A timeout is local. A deadline is a contract.
That distinction matters once an AI agent fans work out to queues, workers, browser sessions, MCP servers, or child agents. If every hop invents its own timeout, the system can keep doing work after the user-visible request has already expired. Worse, a retry may outlive the approval, credential lease, or business deadline that made the original call valid.
The practical fix is to propagate one absolute deadline through the execution graph and enforce it at every boundary.
The failure mode
Suppose a request starts at 12:00:00 with a 30-second budget:
- The orchestrator waits 30 seconds.
- A queue adds 5 seconds of delay.
- A worker starts with its own 30-second timeout.
- A browser tool retries twice with 10 seconds each.
The user sees a timeout at 12:00:30, but the worker and browser can continue until 12:01:05. That is not just wasted compute. A late tool call can send a message, mutate a record, or use a credential after the request is no longer authorized.
A local timeout answers how long will this process wait? A propagated deadline answers after what instant must every hop stop starting new work?
Use an absolute deadline, not a chain of durations
At ingress, calculate one deadline using a monotonic clock. Pass the deadline, rather than the remaining duration, to every child operation.
from dataclasses import dataclass
import time
@dataclass(frozen=True)
class RunContext:
run_id: str
deadline_ns: int
policy_version: str
credential_version: str
def remaining_seconds(ctx: RunContext) -> float:
return max(0.0, (ctx.deadline_ns - time.monotonic_ns()) / 1_000_000_000)
def can_start(ctx: RunContext, reserve_seconds: float = 0.0) -> bool:
return remaining_seconds(ctx) > reserve_seconds
The wall-clock timestamp is useful for logs, but elapsed-time decisions should use a monotonic clock. Wall clocks can jump because of NTP corrections, VM suspension, daylight-saving changes, or operator adjustments.
A child must not replace the parent deadline with a fresh timeout:
def child_budget(ctx: RunContext, expected_seconds: float) -> float:
remaining = remaining_seconds(ctx)
return min(expected_seconds, remaining)
If the remaining budget is zero, the child should be rejected before dispatch. Do not start a call and hope the provider notices the deadline.
Make deadline checks part of dispatch
Check the contract at each side-effect boundary, not only around the top-level request:
- Before enqueue: reject work whose deadline is already expired.
- Before claim: a worker must not claim expired work.
- Before tool dispatch: recheck deadline, policy version, approval, and credential version atomically.
- Before retry: calculate the retry reservation, including backoff and provider timeout.
- Before outbound delivery: use a separate delivery deadline when the notification is still useful after execution expires.
- After an ambiguous result: record UNKNOWN and reconcile with the provider instead of blindly retrying.
The fourth check is commonly missed. A retry with a 2-second provider timeout is unsafe when only 500 milliseconds remain. Backoff is work too, and a retry reservation should include it.
Separate execution and delivery deadlines
A user request can expire while its completion notification is still worth sending. Model those as separate contracts:
{
"execution_deadline": "monotonic:...",
"delivery_deadline": "monotonic:...",
"max_attempts": 2,
"effect_key": "run-123:completion:v1"
}
Execution expiry means do not begin another tool or model step. Delivery expiry means do not send this notification after it is stale. Keeping the ledgers separate prevents a failed notification from causing the business action to run again.
For a hosted, always-on OpenClaw runtime, managed OpenClaw hosting on Ampere is one deployment option to evaluate. It does not define deadline semantics, cancel already-dispatched side effects, or remove the need for idempotency and credential checks.
Test clock and queue failures deliberately
A useful test matrix should include:
| Scenario | Expected result |
|---|---|
| Queue delay consumes the full budget | Worker refuses to claim |
| Worker clock is ahead | Monotonic deadline still governs local enforcement |
| Wall clock jumps backward | No extra execution time is created |
| Retry reservation exceeds remaining budget | Retry is rejected |
| Deadline expires during provider call | Outcome is recorded, not guessed |
| Execution expires but delivery remains valid | Completion is delivered once, if still useful |
| Worker restarts after expiry | Recovery does not resurrect the run |
| Duplicate dispatch races with expiry | One effect key and provider reconciliation decide the result |
Inject delays at enqueue, claim, pre-dispatch, provider response, and delivery. Record the deadline, remaining budget, clock source, policy version, credential version, and effect key in the evidence trail. Without those fields, a log line saying timed out is not enough to explain whether the work never started, was cancelled, or completed ambiguously.
A compact review checklist
Before shipping an agent workflow, verify:
- One ingress deadline is propagated to every child operation.
- Duration budgets are derived from the deadline, never reset at a hop.
- Monotonic time is used for enforcement and wall time only for display.
- Queue claim and side-effect dispatch both recheck expiry.
- Retries reserve backoff plus execution time.
- Execution and outbound delivery have separate state machines.
- Expired or ambiguous effects use stable idempotency keys and reconciliation.
- Restarts cannot resurrect work past its deadline.
- Tests inject queue delay, clock jumps, provider ambiguity, and duplicate races.
The model can choose the next action, but the runtime must decide whether there is still time and authority to take it. A deadline becomes real only when every queue, worker, tool, and delivery path treats it as the same contract.
Top comments (0)