An OpenClaw deployment can be healthy and still be impossible to recover.
The dangerous assumption is that “the process starts” means “the agent is back.” A real recovery also needs its state, credentials, schedules, browser profile, queued work, and delivery records. If any of those live only on the original machine, a host failure turns restart into archaeology.
This article gives you a small, repeatable rebuild drill. It is designed for a single always-on OpenClaw instance, but the same boundaries apply to other long-running agent runtimes.
Define the recovery contract first
Write down what must be true after a rebuild:
- The agent starts with the expected version and configuration.
- Persistent state is present and readable.
- Credentials are restored without printing their values.
- Scheduled jobs are not silently duplicated.
- In-flight work is either resumed from a checkpoint or marked for reconciliation.
- Browser sessions are either restored intentionally or treated as expired.
- Delivery records let you distinguish planned, executed, and delivered.
The last two points matter most. A process supervisor can restart a process, but it cannot tell whether a message was sent before the crash or whether a browser action reached its target.
Inventory state by recovery consequence
Do not start with a backup command. Start with an inventory:
| State | Example | Recovery treatment |
|---|---|---|
| Rebuildable | container image, lockfile, IaC | Recreate from source control |
| Durable | SQLite database, checkpoints, run ledger | Back up and restore atomically |
| Secret | API keys, session cookies | Restore through a secret store; rotate if exposed |
| Ephemeral | PID files, temp files, caches | Delete and regenerate |
| Ambiguous | browser profile, partially sent job | Restore only with an explicit reconciliation rule |
For each path, record the owner, backup location, encryption status, and restore test date. If you cannot answer those four questions, the path is not part of a recovery plan yet.
Use a manifest instead of a vague backup
A manifest makes the drill inspectable. Keep it beside your deployment configuration, not inside a secret dump:
# recovery-manifest.yml
runtime_version: pin-in-source-control
persistent_paths:
- /srv/openclaw/state
- /srv/openclaw/checkpoints
secret_names:
- OPENCLAW_PROVIDER_KEY
- DELIVERY_SIGNING_KEY
reconcile:
queued_work: replay_from_checkpoint
unknown_delivery: hold_for_review
browser_session: expired_until_reauthenticated
health:
- state_readable
- secret_present
- scheduler_disabled_until_verified
- outbound_delivery_tested
The exact paths will vary. The useful part is that the policy is explicit and reviewable.
Rebuild on a clean host
Create a temporary host or isolated VM. Do not use the original machine as the restore target. The test should prove that your deployment does not depend on undocumented local state.
A minimal sequence looks like this:
# Fetch the pinned deployment definition
git clone https://example.invalid/your-openclaw-deployment.git
cd your-openclaw-deployment
git checkout <known-good-commit>
# Install exact runtime dependencies
./scripts/install-runtime.sh
# Restore durable state into a staging directory
./scripts/restore-state.sh ./recovery-archive ./staging-state
./scripts/verify-state.sh ./staging-state
# Start with scheduling and outbound delivery disabled
OPENCLAW_SCHEDULER_ENABLED=false \
OPENCLAW_DELIVERY_ENABLED=false \
./scripts/start-openclaw.sh --state ./staging-state
The example.invalid URL is intentionally a placeholder. Replace it with your own repository. The important property is the order: pin, restore, verify, then start in a non-destructive mode.
For an always-on deployment where you do not want to own the host lifecycle, managed OpenClaw hosting on Ampere can be a relevant option to evaluate. That does not remove the need for a state manifest or restore test. Hosting is not a backup policy.
Reconcile before enabling side effects
After startup, inspect the run ledger and compare it with the last known snapshot:
- Find work whose execution state is started but not completed.
- Find deliveries whose status is unknown.
- Check whether the scheduler created new jobs while the old host was still considered alive.
- Verify the restored credential identity without exposing the secret.
- Run one idempotent test action against a sandbox destination.
- Only then enable scheduling and production delivery.
A useful state machine is:
planned -> started -> executed -> delivered
-> failed
-> unknown -> held_for_review
Never turn unknown directly into retry. A retry may be correct, but only after checking the destination or using an idempotency key. Otherwise recovery can create the duplicate side effect you were trying to avoid.
Failure-injection checklist
A rebuild drill is not complete until you simulate failures:
- Delete the restored checkpoint and confirm startup fails safely.
- Restore a credential with the wrong identity and confirm outbound delivery stays disabled.
- Stop the process after execution but before delivery acknowledgement.
- Start the scheduler twice and confirm only one lease is active.
- Corrupt one state file and verify the archive is rejected atomically.
- Restore an old browser profile and confirm the runtime treats it as expired.
- Replay the same request key and confirm the side effect is not duplicated.
Record the expected result and the observed result. A green health endpoint is not evidence that these cases are handled.
The 15-minute operator test
Once a month, have someone follow the runbook without asking the original operator questions. Time these checkpoints:
- clean host provisioned
- state verified
- runtime started in safe mode
- credentials verified
- unknown work reconciled
- sandbox delivery completed
- production scheduling enabled
If the runbook requires tribal knowledge, update the manifest or scripts. Do not solve the gap with a longer paragraph in an incident document.
The practical definition of portable
An OpenClaw deployment is portable when a clean host can reconstruct the runtime and an operator can explain every piece of work left around the failure boundary.
That is a stronger standard than “the service is running,” but it is also testable. Pin the runtime. Separate durable, secret, and ephemeral state. Start in safe mode. Reconcile unknown work. Prove the rebuild periodically.
If you run OpenClaw in production, the follow is for you if you want more concrete control-plane tests for state, identity, recovery, and delivery rather than another list of model capabilities.
Top comments (0)