Your AI Agent Backup Is Not Recovery Until You Test a Restore

A backup job can be green while your agent is still unrecoverable.

That happens when the backup contains files but not the state that makes those files meaningful: the database schema, pending work, credential references, tool configuration, or the exact version needed to interpret them.

For an AI agent, “we have a snapshot” is not a recovery plan. A recovery plan answers a narrower question:

Can I rebuild the agent in a clean environment, prove what work was in flight, and resume without losing or duplicating an external side effect?

This post shows a small restore drill you can adapt to an OpenClaw-style always-on agent, a coding agent worker, or any service that stores durable state locally.

1. Define the recovery boundary

Start by writing down what must survive. Do not begin with the backup tool.

Item	Example	Recovery question
Configuration	model, tool, and schedule config	Can a clean process load the same policy?
Durable state	SQLite database or event log	Can the agent identify completed and pending work?
Credentials	references to secret-manager entries	Can it reconnect without copying raw secrets into an image?
Workspace	checked-out repository or artifacts	Can it reproduce the inputs for the run?
Delivery state	outbound message or webhook record	Can it distinguish sent from merely attempted?
Version	image digest and migration version	Can the restored state be interpreted safely?

These are separate failure domains. A container image backup does not automatically preserve a mounted volume. A database backup does not automatically preserve the delivery receipt held by another service. A secret reference does not prove that the restored identity has permission to resolve it.

2. Use a restore manifest

Make the backup self-describing. The manifest should be stored beside the backup, not only in an operator’s memory.

{
  "backup_id": "agent-2026-08-19T090000Z",
  "image_digest": "sha256:replace-with-real-digest",
  "state_schema": 7,
  "state_snapshot": "state.sqlite",
  "workspace_commit": "replace-with-commit",
  "credential_refs": ["agent/runtime", "agent/outbound-webhook"],
  "last_event_id": "evt_0189",
  "delivery_cursor": "delivery_0042",
  "created_at": "2026-08-19T09:00:00Z"
}

The values above are examples, not credentials. In a real system, never put API keys, cookies, browser profiles, or bearer tokens in this manifest. Store references and verify access separately.

The important fields are the cursors. “Last event” and “last delivery” are not interchangeable. An agent may have committed a task result locally while its outbound notification was still pending.

3. Restore into an empty environment

A restore drill is invalid if it reuses the original process, filesystem, or credentials without proving the boundary.

Use a disposable environment with:

• a new container or virtual machine
• a new service identity with the minimum required permissions
• a copied backup, mounted read-only first
• outbound delivery pointed at a test sink
• network access limited to the dependencies required for the drill

A minimal sequence might look like this:

set -euo pipefail

export BACKUP_DIR=./restore-drill/agent-2026-08-19T090000Z
export RESTORE_DIR=$(mktemp -d)

cp "$BACKUP_DIR/manifest.json" "$RESTORE_DIR/"
cp "$BACKUP_DIR/state.sqlite" "$RESTORE_DIR/"

sqlite3 "$RESTORE_DIR/state.sqlite" 'PRAGMA integrity_check;'
# Start the exact recorded image with outbound delivery disabled.
# Run migrations in check-only mode before allowing the worker to start.

The first boot should be read-only or dry-run where possible. This catches a dangerous class of failure: a restore process that starts consuming a queue or sending messages before an operator has verified the recovered cursors.

4. Test the invariants, not just process liveness

A passing health endpoint proves very little. Check invariants that correspond to real user-visible failure:

-- No task may be both completed and pending.
SELECT task_id
FROM task_attempts
GROUP BY task_id
HAVING SUM(status = 'completed') > 0
   AND SUM(status IN ('queued', 'running')) > 0;

-- Every delivery marked sent must have a durable provider receipt.
SELECT delivery_id
FROM deliveries
WHERE status = 'sent' AND provider_receipt IS NULL;

-- Every running task must have an owner and a lease timestamp.
SELECT task_id
FROM task_attempts
WHERE status = 'running'
  AND (worker_id IS NULL OR lease_until IS NULL);

Your test should fail loudly if these queries return rows. An empty result is evidence for that check only; it is not proof that the whole restore is correct.

Then inject a known test task and verify its complete lifecycle:

1. The restored worker claims it once.
2. A forced process termination leaves an observable intermediate state.
3. A replacement worker resumes or retries according to policy.
4. The test delivery reaches the sink exactly once, or is explicitly marked duplicate-safe.
5. The final state is visible after restarting the replacement worker again.

Do not use a real customer notification or production credential for this test.

5. Test the crash points separately

At minimum, exercise these boundaries:

• after the task is committed, before the delivery is attempted
• after the delivery request is sent, before the provider receipt is stored
• after the receipt is stored, before the task is marked complete
• during a schema migration
• after restoring an older backup with a newer binary

The second case is the one many systems get wrong. If the network request succeeded but the process died before recording the response, a blind retry can duplicate the side effect. The safe design is not “retry more.” It is an idempotency key or a provider-side deduplication contract, plus a durable record of the attempt.

6. Set a recovery acceptance checklist

A restore drill should produce a small signed or append-only report containing:

• backup identifier and image digest
• schema migration result
• state integrity result
• credential-reference checks, without secret values
• queue and delivery cursor comparison
• test task ID and delivery receipt
• time to restore a usable dry-run worker
• unresolved differences and an owner

If the report cannot answer what the agent might repeat, the drill is incomplete.

For agents that need to run continuously, managed always-on agent hosting on Ampere can be useful when the operational problem is keeping the runtime and its persistent state together. It does not remove the need to define credential boundaries, test restores, or make outbound actions idempotent. Those remain application responsibilities.

What to do this week

1. Pick one agent with a real persistent state store.
2. Write its recovery boundary and record the image, schema, workspace, identity references, and delivery cursor.
3. Restore it into an empty environment with outbound delivery disabled.
4. Run integrity and invariant checks before starting the worker.
5. Inject one harmless test task and kill the process at two different crash points.
6. Keep the restore report with the backup.

A backup is a copy. Recovery is a verified behavior under failure. For an agent that can change code, call tools, or send messages, that distinction is the difference between restarting a service and safely resuming work.

What does your restore drill verify separately: execution state, outbound delivery state, or both?