A rolling deploy can be perfectly healthy at the process level and still corrupt an agent run.
The dangerous window is not startup. It is the tool call that was accepted by the old worker while the deploy moved traffic to the new one. If the old worker is killed without a protocol, the replacement has to guess whether the side effect happened.
That is why graceful shutdown for an agent needs more than SIGTERM and a health check. It needs a drain contract.
The drain contract
Give each worker an explicit lifecycle:
ACCEPTING -> DRAINING -> QUIESCED -> STOPPED
A worker in ACCEPTING may claim new runs. A worker in DRAINING rejects new claims but finishes work already inside a safe boundary. QUIESCED means there are no owned operations left, or every remaining operation has moved to a durable reconciliation queue. Only then should the process exit.
The important detail is that “no active HTTP requests” is not the same as “no active agent work.” A tool call can outlive the request that started it, and a browser or provider can still be processing a side effect after the worker disappears.
Separate run draining from effect draining
Track these states independently:
| Thing | Safe-to-stop condition |
|---|---|
| Run execution | No step is executing, or its checkpoint is durable |
| Tool dispatch | No call is between intent and provider outcome |
| Browser session | No mutation is in flight, or the outcome is reconciled |
| Outbound delivery | Every message has a provider ID or an UNKNOWN record |
Do not make shutdown wait forever for an external provider. Set a bounded drain deadline. When the deadline expires, persist the operation as UNKNOWN, release the worker lease, and let a reconciler query the provider or apply a documented manual check.
This is the same reason a run lease needs fencing rather than just a timeout: a timeout detects waiting, but it does not prove that an old worker has stopped acting.
A minimal shutdown sequence
- Flip the worker state to DRAINING in durable storage.
- Stop accepting new queue claims. Enforce this at the claim transaction, not only in the router.
- Wait for active steps to reach a checkpoint or a terminal outcome.
- For each in-flight effect, write or update an operation record with its stable idempotency key.
- At the deadline, mark unresolved operations UNKNOWN and enqueue reconciliation.
- Publish QUIESCED only after the worker owns no claims or leases.
- Terminate the process and verify that no stale fencing token is accepted by downstream tools.
A useful invariant is:
A worker may exit only when it cannot start new work and every old operation is either terminal or durably UNKNOWN.
Test the failure, not just the happy path
A deploy test should deliberately pause the worker at each boundary:
- after the queue claim but before the checkpoint
- after intent is recorded but before provider dispatch
- after provider dispatch but before the response is stored
- after the browser mutation but before the result is acknowledged
- after the drain deadline but before the old process exits
For every pause, kill the old worker, start a replacement, and check:
- Was the old claim fenced?
- Did the replacement resume from a durable checkpoint?
- Did an ambiguous effect become UNKNOWN instead of being blindly retried?
- Can reconciliation find the provider-side result?
- Is the same idempotency key used on every retry?
Log drain_started_at, drain_deadline_at, quiesced_at, unknown_count, and stale_claim_rejections. These signals tell you whether deploys are actually reducing risk or merely shortening process downtime.
Hosting does not create the contract
An always-on runtime can make worker lifecycle management easier to operate, but it does not define your drain semantics. If you need a managed place to run an OpenClaw or browser-automation worker, managed always-on agent hosting on Ampere is one option to evaluate. Keep the drain ledger, fencing checks, credential boundaries, and reconciliation logic in your application.
The practical goal is not zero interruption. It is a deploy where every interruption has a known state, every stale worker is rejected, and no operator has to guess whether a tool call happened.
If you build agent runtimes, follow for concrete failure-injection drills and control-plane patterns that survive the demo.
Top comments (0)