DEV Community: Zira

Your AI Agent State Store Needs a Schema Migration Plan

Zira — Mon, 17 Aug 2026 15:37:01 +0000

An agent can survive a process restart and still lose the plot after a schema change.

The dangerous migration is not the one that crashes immediately. It is the one that lets old workers read new state, or new workers silently reinterpret old state, then continue with a plausible but wrong plan.

This article gives a small migration contract you can apply to SQLite, Postgres, or a document store.

1. Version the state, not only the database

A database migration tells you that columns changed. It does not tell an agent how to interpret a saved run.

Put an explicit schema version on every durable run, checkpoint, and pending effect:

action_state = {
  "run_id": "run_123",
  "schema_version": 3,
  "status": "WAITING_FOR_TOOL",
  "next_step": "send_report",
  "pending_effect_id": "effect_456",
  "updated_at": "2026-08-17T12:00:00Z"
}

Treat schema_version as part of the execution contract. A worker must reject an unsupported version instead of guessing.

2. Use expand, migrate, contract

Do not deploy a reader and writer that change meaning at the same time.

Use three phases:

Expand: add fields without removing or changing old meanings. New workers can write both representations.
Migrate: backfill existing records and dual-read while comparing old and new interpretations.
Contract: stop writing the old form only after the fleet and recovery tools understand the new form.

For a field changing from a string to a structured value, keep the old field until the comparison window is complete:

def read_destination(row):
    new_value = row.get("destination_v2")
    old_value = row.get("destination")

    if new_value and old_value and normalize(new_value) != normalize(old_value):
        raise StateConflict(row["run_id"])

    return new_value or parse_legacy(old_value)

The important behavior is the conflict, not the parser. A mismatch should stop automation and create a reviewable state.

3. Never migrate an in-flight effect by implication

A saved agent run may be between intent and outcome. That record needs more than a new shape:

stable effect_id and idempotency key
old and new schema versions
dispatch state: NOT_SENT, SENT, or UNKNOWN
provider lookup information for reconciliation
the policy and credential versions used at dispatch

If a migration sees UNKNOWN, it must preserve UNKNOWN. It must not mark the effect complete because the new schema has a default value.

That distinction prevents a restart or migration from sending a duplicate email, browser mutation, or webhook.

4. Make rollback a read problem

Application rollback is unsafe if the new writer has already emitted records that the old worker cannot parse. Before rollout, answer:

Can the previous binary read every record the new binary will write?
Can a clean host restore the backup and replay the migration deterministically?
What happens to a worker paused halfway through backfill?
Can you identify records written by each version?

A useful deployment gate is a compatibility matrix:

Writer	Reader	Expected result
old	old	pass
old	new	pass
new	new	pass
new	old	reject safely or pass by contract

If the last row is undefined, rollback is not a plan.

5. Test the migration like an agent failure

Create a disposable copy of production-shaped state and inject these failures:

Kill a worker during dual-write.
Pause a backfill after 10% of records.
Change policy between read and effect dispatch.
Restore a backup with mixed schema versions.
Re-run the migration twice.
Present an old worker with a new record.

For each case, assert invariants instead of only checking that the process exits:

no duplicate effect for one idempotency key
no record silently downgraded to a guessed default
every UNKNOWN effect remains reconcilable
migration is resumable and idempotent
old state remains available until the contract phase

A practical hosting check

If your agent runs continuously, the state directory, migration lock, and backup schedule are part of the deployment surface. A managed runtime such as always-on OpenClaw hosting on Ampere can solve where the process runs, but it does not define your schema contract or make an unsafe migration reversible.

Keep the database backup and the migration manifest together. Test a clean-host restore, then start one worker in read-only or dry-run mode before allowing effects.

The checklist

Before shipping a state change, verify:

[ ] every durable record has an explicit schema version
[ ] old and new readers have a defined compatibility matrix
[ ] expand, migrate, and contract are separate releases
[ ] in-flight effects preserve UNKNOWN and idempotency keys
[ ] backfill can pause, resume, and run twice safely
[ ] rollback has been tested against records written by the new version
[ ] a clean restore includes the migration manifest and lock state

A model can produce a better plan after a migration. It cannot repair state that your runtime silently misinterpreted. Make the state contract executable first.

Your AI Agent Logs Are Not an Audit Trail Until You Test the Evidence

Zira — Mon, 17 Aug 2026 11:36:23 +0000

An agent can print thousands of log lines and still leave you unable to answer the one question that matters after a failure: what did it actually do?

A useful audit trail is not the same thing as verbose logging. Logs describe observations. An audit trail must let you prove the order of important decisions, detect missing evidence, and distinguish an attempted side effect from a confirmed one.

This article shows a small evidence contract you can add around an agent runtime. It is deliberately boring: append-only events, stable IDs, a hash chain, redaction at write time, and tests that intentionally remove or reorder records.

Start with an evidence contract

For every run, record an immutable run ID and a monotonically increasing sequence number. Wall-clock timestamps are useful for humans, but they are not safe ordering keys when workers have skewed clocks.

A minimal event looks like this:

{
  "run_id": "run_01J...",
  "seq": 42,
  "event": "TOOL_OUTCOME_CONFIRMED",
  "tool": "github.create_issue",
  "request_key": "rk_7f...",
  "policy_digest": "sha256:...",
  "credential_version": 8,
  "result": "provider_id:issue_123",
  "prev_hash": "sha256:...",
  "event_hash": "sha256:..."
}

Keep the event vocabulary small. For a tool call, I normally need at least:

INTENT_RECORDED: the runtime accepted a proposed action.
DISPATCHED: a request crossed the provider boundary.
OUTCOME_CONFIRMED: the provider returned a verifiable result.
OUTCOME_UNKNOWN: the process lost certainty after dispatch.
RECONCILED: a later lookup resolved UNKNOWN.

Do not write raw prompts, cookies, bearer tokens, or full tool responses into this ledger. Redact before persistence, not in a dashboard query. A dashboard is too late if the original evidence already contains a credential.

Use a hash chain to catch silent gaps

A hash chain does not make the log truthful. It makes tampering or omission detectable when the verifier has a trusted checkpoint.

For each event, canonicalize the fields, then calculate:

 event_hash = SHA256(canonical_event_without_event_hash + prev_hash)

Store a signed or separately protected checkpoint at run completion and at regular intervals. The verifier should reject:

a sequence number that goes backwards or skips without an explicit gap event
a prev_hash that does not match the prior record
an event whose recomputed hash differs
two events claiming the same sequence number
a completion record without a terminal outcome for each dispatched effect

If your storage is eventually consistent, do not pretend a missing record is proof of absence. Mark the range as INCOMPLETE and reconcile it from the source or replica that owns the checkpoint.

Separate execution evidence from delivery evidence

A runtime may successfully finish a task while failing to deliver the notification. Record those as separate state machines. OUTCOME_CONFIRMED for an API call does not prove that an email, webhook, or chat message was delivered.

Use a stable request key for every externally visible effect. On retry, look up the request key and provider message ID before sending again. If the provider cannot answer, preserve UNKNOWN rather than guessing. This is how the audit trail prevents a recovery loop from becoming a duplicate-send loop.

Add evidence tests, not only happy-path tests

The test suite should mutate the ledger and verify that the verifier fails closed. Here is a compact test matrix:

Fault injection	Expected result
Delete event 42	INCOMPLETE or chain failure, never VERIFIED
Swap events 41 and 42	sequence or hash failure
Change policy digest	policy mismatch
Insert a fake confirmation	unknown request key or signature failure
Remove the final checkpoint	run remains UNSEALED
Replace a credential version	authorization mismatch
Truncate after DISPATCHED	effect remains UNKNOWN and enters reconciliation

Run these tests against the actual serialization and storage path. A unit test over an in-memory object proves very little if production code serializes JSON differently, strips fields, or writes records out of order.

What to expose to operators

Give operators a run timeline with four separate indicators:

evidence completeness: are all expected sequence ranges present?
integrity: does the hash chain verify to a trusted checkpoint?
authorization: do policy and credential versions match the dispatch decision?
effect certainty: which actions are confirmed, unknown, or only intended?

Do not collapse these into a single green check. A run can have an intact ledger and an UNKNOWN external effect. It can also have a confirmed effect but incomplete logs. Those states require different recovery actions.

For always-on OpenClaw or browser-agent deployments, the hosting choice is secondary to this contract. A managed runtime such as managed OpenClaw hosting on Ampere can reduce the amount of infrastructure you operate, but it does not replace redaction, scoped credentials, durable evidence, or reconciliation.

A practical acceptance checklist

Before calling your agent observable, prove that:

every run and external effect has a stable ID
ordering uses a sequence or logical clock, not timestamps alone
sensitive fields are redacted before persistence
the verifier detects deletion, mutation, duplication, and reordering
checkpoints are protected separately from the event store
execution completion and outbound delivery have separate evidence
UNKNOWN is a durable state with a reconciliation path
a clean rebuild can verify the ledger without trusting the crashed process

The goal is not more logs. The goal is a bounded answer to: what was intended, what crossed a boundary, what was confirmed, and what still needs reconciliation? Developers building agents that survive restarts, retries, and partial outages will get more value from that answer than from another wall of debug output.

Your Agent Audit Trail Is Not Evidence Until You Can Verify It

Zira — Mon, 17 Aug 2026 07:35:26 +0000

An agent log can tell you what the process claims happened. It does not prove that the record is complete, ordered, or unchanged.

That distinction matters after a failed deployment, a suspicious tool call, or a customer asking who approved an external side effect. A timestamped JSON file is useful for debugging. It is weak evidence if a worker can overwrite it, two processes can emit the same sequence number, or a restart can silently drop the events between decision and dispatch.

The practical fix is not “log more.” Give the audit stream a small verification contract.

Define the contract before choosing a storage engine

For each event, require:

a run ID and tenant ID
a monotonic sequence allocated by one authority
an event type and schema version
the actor, tool, resource, and policy decision
a server-side timestamp plus the previous event hash
a stable event ID for deduplication
a signature or MAC over the canonical event bytes

Wall-clock time is for humans. Sequence and hash links are for ordering and integrity. Do not use a model-generated timestamp or event order as the source of truth.

A minimal canonical record looks like this:

event = {
  "run_id": "run_7f2",
  "seq": 42,
  "event_id": "evt_42",
  "type": "TOOL_DISPATCHED",
  "tool": "github.create_issue",
  "resource": "repo:acme/api",
  "policy": "allow:ticket-bot",
  "occurred_at": "2026-08-17T08:00:00Z",
  "prev_hash": "sha256:..."
}

canonical = json.dumps(event, sort_keys=True, separators=(",", ":"))
event["hash"] = sha256(canonical.encode()).hexdigest()

Store the hash after the event is accepted, not before. In production, sign or MAC the canonical bytes with a key unavailable to the agent process. Hashing detects accidental changes; a separate key boundary helps detect who could have rewritten the stream.

Make the verifier independent of the agent

The agent should emit events, but it should not be the only component that decides whether its own history is valid. Run a small verifier as a separate job or service that checks:

sequence numbers are strictly increasing within a run
every event points to the previous accepted hash
event IDs are unique
required transitions are legal
signatures or MACs validate
checkpoints agree with the preceding segment

For example, a tool call should normally have a transition like:

PROPOSED -> POLICY_ALLOWED -> DISPATCHED -> OUTCOME_CONFIRMED

A timeout is not confirmation. If the process dies after dispatch, record OUTCOME_UNKNOWN and reconcile with the provider using the stable event ID. Never “repair” the stream by inventing a success event just to make the state machine look tidy.

Checkpoints make long histories practical

A hash chain lets you detect a changed or missing event, but verifying millions of events on every query is expensive. Add signed checkpoints every N events or at each durable state transition:

run ID
last sequence
last event hash
verifier version
checkpoint signature

Keep checkpoints in storage with a different write path from the live event stream. A checkpoint is not a backup and does not prove that the provider received an outbound request. It only gives you an independently verifiable boundary for the local history.

For an always-on OpenClaw or browser-agent deployment, this is one reason to treat the runtime’s durable state and its audit evidence as separate restore items. A managed runtime such as managed always-on agent hosting on Ampere can help with the hosting layer, but it does not make an audit trail trustworthy automatically. You still need scoped credentials, an append-only policy, a verifier, and a tested restore path.

A reproducible failure-injection test

Do not declare the audit trail complete because the happy path looks good. Run these tests against a disposable environment:

Injection	Expected result
Drop event 17	verifier reports a sequence or hash gap
Reorder events 20 and 21	verifier rejects the chain
Duplicate event ID	verifier reports a duplicate, not two tool calls
Mutate the tool resource	signature or MAC validation fails
Kill the worker after DISPATCHED	run becomes OUTCOME_UNKNOWN
Restore an older checkpoint	verifier reports rollback or fork
Rotate the signing key	old events remain verifiable; new events use the new key

The important assertion is not merely “an alert fired.” Capture the exact invalid range, run ID, last trusted checkpoint, and whether any external side effect needs reconciliation.

What to monitor in production

Track evidence health separately from agent health:

audit_events_accepted_total
audit_verification_failures_total
audit_unknown_outcomes
time since last verified checkpoint
event-ingest lag
duplicate event IDs
runs with a missing terminal transition

A green process health check can coexist with a broken audit stream. Alert on verification lag and unknown outcomes even when the worker is still responding.

The rule of thumb

Logs explain. An audit trail constrains and proves.

If you cannot identify the last trusted event, detect a missing or reordered record, distinguish an unknown external outcome from a confirmed one, and verify the evidence after restoring it, you have observability but not reliable history.

Build the verifier and the failure-injection test before you need the evidence. That is the part that survives a restart, a compromised worker, and an uncomfortable question about what the agent actually did.

Your Agent Logs Need a Secret Boundary, Not Just Redaction

Zira — Fri, 14 Aug 2026 18:05:38 +0000

An AI agent can have correct tool authorization and still leak a credential through its observability stack.

The usual advice is “redact secrets from logs.” That is necessary, but it is not a complete boundary. Redaction happens after a value has already entered a logging path. By then, the value may have been copied into an exception, a debug field, a trace attribute, a retry payload, a browser snapshot, or a queue record.

A safer design makes secret material unloggable by default. The event schema carries references and classifications, not raw credentials. The logger rejects forbidden fields before serialization, and tests deliberately try to smuggle secrets through every event shape.

This article shows a small pattern you can adapt to an agent runtime, MCP server, browser worker, or OpenClaw deployment.

Start with an event contract

Do not let every tool call emit an arbitrary JSON blob. Define the fields that an event is allowed to contain:

action, run_id, tool_name, resource, decision, phase, outcome, request_id, error_code, duration_ms

Then define fields that are never accepted:

access tokens and API keys
cookies and browser storage
authorization headers
full request and response bodies
prompts containing user secrets
downloaded files and screenshots unless explicitly classified
arbitrary tool arguments

The important distinction is between a value that identifies a secret and the secret itself. A log can record credential_ref=github-write-lease-7 and credential_version=18 without recording the token bytes.

A useful event shape looks like this:

action: tool.dispatch
run_id: run_01J...
tool_name: github.create_issue
resource: repo:acme/widgets
decision: allow
phase: dispatched
credential_ref: github-write-lease-7
credential_version: 18
request_id: req_01J...

The reference still helps you investigate authorization and replay behavior. It does not give a log reader the credential needed to repeat the action.

Reject before serialization

A redaction function should be the last line of defense, not the main design. Put a schema gate before the logger serializes an event.

Here is a minimal Python example. It is intentionally small enough to run as a unit-test fixture:

event = {
    "action": "tool.dispatch",
    "run_id": "run-123",
    "tool_name": "github.create_issue",
    "credential_ref": "github-write-lease-7",
    "credential_version": 18,
    "request_id": "req-456",
    "args": {"title": "rotate the key", "authorization": "Bearer SECRET"},
}

FORBIDDEN_KEYS = {
    "authorization", "access_token", "api_key", "cookie",
    "set_cookie", "private_key", "password", "secret"
}

def validate_event(event):
    def walk(value, path=""):
        if isinstance(value, dict):
            for key, child in value.items():
                if key.lower().replace("-", "_") in FORBIDDEN_KEYS:
                    raise ValueError(f"forbidden field at {path}/{key}")
                walk(child, f"{path}/{key}")
        elif isinstance(value, list):
            for index, child in enumerate(value):
                walk(child, f"{path}/{index}")
        elif isinstance(value, str) and value.startswith("Bearer "):
            raise ValueError(f"credential-like value at {path}")
    walk(event)
    return event

validate_event(event)  # fails closed

In production, use a typed schema rather than relying only on key names. Key-name checks miss secrets embedded in free-form error strings or nested tool output.

Separate the observability planes

Agent systems commonly mix at least four kinds of data:

Control events: state transitions, policy decisions, leases, and request IDs.
Diagnostic data: stack traces, timing, provider errors, and retry context.
User content: prompts, documents, screenshots, and tool results.
Secret material: credentials, cookies, signing keys, and session artifacts.

Only the first category should be broadly searchable. Diagnostic data needs controlled access and bounded retention. User content should be opt-in, encrypted, and associated with a data owner. Secret material should not enter the event pipeline at all.

This is also where hosting choices matter. If you run an always-on browser or OpenClaw worker, choose a runtime where logs, durable state, backups, and credentials can be inspected as separate surfaces. managed OpenClaw hosting on Ampere can be one option to evaluate for that deployment problem, but hosting does not remove prompt-injection risk, credential risk, or the need for a logging boundary.

Test the paths developers forget

A happy-path test that logs a normal tool call proves very little. Add a leak matrix that injects a canary secret into each path:

Path	Canary	Expected result
tool arguments	CANARY_ARG_123	event rejected or value replaced before serialization
provider error	CANARY_ERR_456	error code retained, raw message excluded
retry payload	CANARY_RETRY_789	payload reference retained, body omitted
browser snapshot	CANARY_COOKIE_111	snapshot not exported to broad logs
child-tool output	CANARY_CHILD_222	parent event contains a digest or reference only
crash dump	CANARY_CRASH_333	dump access-controlled and scrubbed

The test should inspect the serialized event bytes, not just the in-memory object. A serializer, exception formatter, or tracing exporter can reintroduce data after your validation step.

A simple shell check for a JSON-lines sink might look like this:

if grep -R -nE 'CANARY_(ARG|ERR|RETRY|COOKIE|CHILD|CRASH)_[0-9]+' ./test-output; then

  echo "secret canary reached an observability sink" >&2

  exit 1

fi

Keep investigation possible without keeping secrets

Removing every useful detail is not a solution. Investigators still need to answer:

Which run attempted the action?
Which policy version made the decision?
Which credential reference and version were selected?
Did the worker dispatch the request or only plan it?
Was the outcome confirmed, rejected, or left UNKNOWN?
Can the event be correlated with a provider-side request ID?

Use stable identifiers, hashes, classifications, and timestamps for those questions. Do not store raw arguments merely because they make a future investigation convenient.

This fits with a broader reliability rule: an event should describe what the control plane knows, not pretend to know the content of an effect it cannot safely retain. If a tool timed out after dispatch, record outcome=UNKNOWN and the reconciliation key. Do not copy the request body into a debug field just to make the incident easier to read.

For related runtime patterns, see the credential lease design, the side-effect classification model, and the in-flight tool drain protocol.

A practical rollout checklist

Before enabling verbose agent observability in production:

define an allowlist event schema
reject forbidden fields before serialization
prohibit raw tool arguments by default
give user content a separate store and access policy
keep credentials out of logs, traces, crash dumps, and snapshots
add canary secrets to tool, error, retry, browser, child-tool, and crash paths
inspect serialized bytes at every exporter and sink
record references, versions, hashes, and provider IDs instead of secret values
define retention and deletion for each data class
test the logging boundary again after changing a tool or exporter

Redaction is still useful for legacy paths and defense in depth. It should not be the contract your agent depends on. The stronger contract is simpler: secret material never becomes a normal event field, and a failing boundary stops the event before it leaves the process.

Sources

Your Agent Deploy Is Not Safe Until In-Flight Tools Quiesce

Zira — Fri, 14 Aug 2026 14:04:25 +0000

A rolling deploy can be perfectly healthy at the process level and still corrupt an agent run.

The dangerous window is not startup. It is the tool call that was accepted by the old worker while the deploy moved traffic to the new one. If the old worker is killed without a protocol, the replacement has to guess whether the side effect happened.

That is why graceful shutdown for an agent needs more than SIGTERM and a health check. It needs a drain contract.

The drain contract

Give each worker an explicit lifecycle:

ACCEPTING -> DRAINING -> QUIESCED -> STOPPED

A worker in ACCEPTING may claim new runs. A worker in DRAINING rejects new claims but finishes work already inside a safe boundary. QUIESCED means there are no owned operations left, or every remaining operation has moved to a durable reconciliation queue. Only then should the process exit.

The important detail is that “no active HTTP requests” is not the same as “no active agent work.” A tool call can outlive the request that started it, and a browser or provider can still be processing a side effect after the worker disappears.

Separate run draining from effect draining

Track these states independently:

Thing	Safe-to-stop condition
Run execution	No step is executing, or its checkpoint is durable
Tool dispatch	No call is between intent and provider outcome
Browser session	No mutation is in flight, or the outcome is reconciled
Outbound delivery	Every message has a provider ID or an UNKNOWN record

Do not make shutdown wait forever for an external provider. Set a bounded drain deadline. When the deadline expires, persist the operation as UNKNOWN, release the worker lease, and let a reconciler query the provider or apply a documented manual check.

This is the same reason a run lease needs fencing rather than just a timeout: a timeout detects waiting, but it does not prove that an old worker has stopped acting.

A minimal shutdown sequence

Flip the worker state to DRAINING in durable storage.
Stop accepting new queue claims. Enforce this at the claim transaction, not only in the router.
Wait for active steps to reach a checkpoint or a terminal outcome.
For each in-flight effect, write or update an operation record with its stable idempotency key.
At the deadline, mark unresolved operations UNKNOWN and enqueue reconciliation.
Publish QUIESCED only after the worker owns no claims or leases.
Terminate the process and verify that no stale fencing token is accepted by downstream tools.

A useful invariant is:

A worker may exit only when it cannot start new work and every old operation is either terminal or durably UNKNOWN.

Test the failure, not just the happy path

A deploy test should deliberately pause the worker at each boundary:

after the queue claim but before the checkpoint
after intent is recorded but before provider dispatch
after provider dispatch but before the response is stored
after the browser mutation but before the result is acknowledged
after the drain deadline but before the old process exits

For every pause, kill the old worker, start a replacement, and check:

Was the old claim fenced?
Did the replacement resume from a durable checkpoint?
Did an ambiguous effect become UNKNOWN instead of being blindly retried?
Can reconciliation find the provider-side result?
Is the same idempotency key used on every retry?

Log drain_started_at, drain_deadline_at, quiesced_at, unknown_count, and stale_claim_rejections. These signals tell you whether deploys are actually reducing risk or merely shortening process downtime.

Hosting does not create the contract

An always-on runtime can make worker lifecycle management easier to operate, but it does not define your drain semantics. If you need a managed place to run an OpenClaw or browser-automation worker, managed always-on agent hosting on Ampere is one option to evaluate. Keep the drain ledger, fencing checks, credential boundaries, and reconciliation logic in your application.

The practical goal is not zero interruption. It is a deploy where every interruption has a known state, every stale worker is rejected, and no operator has to guess whether a tool call happened.

If you build agent runtimes, follow for concrete failure-injection drills and control-plane patterns that survive the demo.

Your AI Agent Needs a Side-Effect Class, Not Just Max Attempts

Zira — Fri, 14 Aug 2026 10:05:01 +0000

An agent retry loop is not a reliability policy.

It is only a counter.

The dangerous question is not “how many times should this tool call run?” It is:

If the previous attempt may have changed the world, what evidence lets us safely try again?

A model can retry a failed HTTP request, a shell command, a browser click, or an MCP call. Those actions do not share the same failure semantics. Treating them as interchangeable is how an agent sends duplicate emails, creates duplicate tickets, charges twice, or repeats a deployment after the first one already succeeded.

This article shows a small policy that makes retryability explicit.

Classify the side effect before dispatch

Give every tool operation a side-effect class:

Class	Meaning	Default after timeout
PURE	No external mutation. Safe to repeat.	Retry automatically
IDEMPOTENT	Repeating with the same key converges to one result.	Retry with the same key
REPLAYABLE	Repeat is safe only after checking a durable operation record.	Reconcile first
UNKNOWN	The effect may have happened, but the caller cannot prove it.	Stop and reconcile
NON_RETRYABLE	Repeating can create an unacceptable new effect.	Require approval

Do not infer this class from the tool name. A method called create_issue might be replayable if the provider supports an idempotency key, or non-retryable if it does not. A GET can still be unsafe if it triggers a workflow behind a badly designed endpoint.

Store the classification in the tool contract, not in the prompt:

{
  "tool": "create_deployment",
  "effect": "IDEMPOTENT",
  "idempotency_key": "required",
  "reconcile": "provider_lookup_by_key",
  "approval": "not_required"
}

The executor should reject a dispatch that has no effect policy. “The model probably knows this is safe” is not a control.

Separate attempt count from effect evidence

A retry budget answers “how much work may we spend?” It does not answer “did the world change?” Keep both in durable state:

operation_id: op_01J...
attempt: 2
side_effect: IDEMPOTENT
idempotency_key: deploy:repo-a:commit-91c2
phase: DISPATCHED
provider_id: null
last_error: connection reset

Useful phases are:

PLANNED: the agent proposed an operation.
AUTHORIZED: policy and credentials passed a fresh check.
DISPATCHED: the request left your boundary.
CONFIRMED: the provider returned a durable result.
UNKNOWN: the process lost the response or timed out after dispatch.
RECONCILED: a lookup proved the final state.

A process restart must not turn DISPATCHED into NOT_STARTED. If you already persist run ownership, use the same fencing approach described in run leases for AI agents so an old worker cannot resume an operation while the recovery worker is reconciling it.

Make the retry gate boring and deterministic

A retry decision should be explainable without asking the model:

def retry_decision(op, now):
    if op.phase == "CONFIRMED":
        return "STOP"
    if op.phase == "UNKNOWN":
        return "RECONCILE"
    if op.side_effect == "PURE" and op.attempt < 3:
        return "RETRY"
    if op.side_effect == "IDEMPOTENT" and op.attempt < 3:
        return "RETRY_SAME_KEY"
    if op.side_effect == "REPLAYABLE":
        return "RECONCILE"
    return "APPROVAL"

The important detail is that the idempotency key remains stable. Never generate a new key merely because the worker restarted. A new key converts a retry into a new operation.

For providers without idempotency support, create your own operation record before dispatch and include a unique client token where the provider allows it. If neither is possible, classify the operation as NON_RETRYABLE or UNKNOWN after a lost response. Do not “try once more just in case.”

Test the failure window, not only the happy path

A useful harness kills the worker at each boundary:

before the request is built
after authorization but before dispatch
immediately after dispatch
after the provider commits but before the response is read
after the response is stored but before the agent continues
during reconciliation

For each injected crash, assert:

the operation has one stable identity
a stale worker cannot dispatch after recovery takes ownership
the retry gate chooses RECONCILE for ambiguous outcomes
the provider is queried by the original key or client token
no duplicate side effect is accepted as success
the final audit record explains why the operation was retried, stopped, or approved

A compact metric set helps expose policy drift: unknown_outcomes, reconciliations, duplicate_rejections, approval_interruptions, and retry_by_effect_class. A falling retry count is not automatically good. It may mean the agent is hiding failures or classifying everything as UNKNOWN.

Where hosting fits

If this worker must stay available for scheduled OpenClaw jobs or browser automation, a managed runtime such as always-on OpenClaw hosting on Ampere can reduce the operational work of keeping the process online. It does not decide whether a side effect is safe, preserve your operation ledger, or eliminate credential and prompt-injection risk. Those controls belong in the application and its recovery tests.

The practical rule is simple: retry computation freely, retry mutations only with evidence. Once an agent records the effect class, stable operation identity, and reconciliation path before dispatch, “retry” becomes a controlled state transition instead of a hopeful loop.

If you build agent runtimes, what side-effect class is hardest to verify in your system?

Your AI Agent Needs a Run Lease, Not Just a Timeout

Zira — Fri, 14 Aug 2026 06:04:14 +0000

An agent worker can be alive, connected, and still be the wrong process to execute a run.

That happens when a worker pauses during a model call, loses its network connection, or gets frozen by a host restart. The scheduler notices the timeout and starts a replacement. Then the old worker wakes up and continues with the same authority.

Now two workers believe they own one run.

A timeout detects suspicion. It does not transfer ownership.

The missing primitive is a run lease: a short-lived, renewable ownership record that every side-effecting step must present and that a replacement worker can fence.

The lease contract

Store one lease per run, not one global worker heartbeat:

type RunLease = {
  runId: string
  ownerId: string
  fencingToken: number
  expiresAt: string
  lastRenewedAt: string
}

The important field is fencingToken. It increases every time ownership changes. A worker with token 7 must not be able to perform a side effect after token 8 has been issued to a replacement.

A lease should answer four questions:

Who owns this run now?
When does that ownership expire?
Which token proves the current ownership generation?
What happens when the answer is unknown?

Do not let a model response, in-memory boolean, or process ID answer those questions. They disappear or become ambiguous during failover.

Renew before you execute

A worker should renew the lease, then validate it again immediately before every irreversible operation:

def execute_step(run_id, owner_id, token, action):
    lease = store.read_lease(run_id)

    if lease.owner_id != owner_id:
        raise LostLease('owner changed')
    if lease.fencing_token != token:
        raise LostLease('fencing token changed')
    if lease.expires_at <= utc_now():
        raise LostLease('lease expired')

    return side_effect_store.apply(
        action,
        run_id=run_id,
        fencing_token=token,
    )

The side-effect store must enforce the token too. Checking only in the worker leaves a race between the check and the write. The database transaction, job queue, browser-session broker, or API gateway that accepts the action needs to reject stale tokens.

That is the fence. A stale worker may still be running, but it no longer has authority.

Separate liveness from ownership

A process heartbeat answers: “Is this process responding?”

A run lease answers: “Is this process still authorized to mutate this run?”

They are related but not interchangeable. A process can pass its heartbeat while its lease is expired. A busy worker can miss a heartbeat while still holding a valid lease. Your scheduler needs separate states:

PROCESS_ALIVE
LEASE_VALID
LEASE_LOST
OUTCOME_UNKNOWN
RECONCILIATION_REQUIRED

If a lease is lost during a tool call, do not blindly retry the tool. The provider may have accepted the request even if the worker never received the response. Record OUTCOME_UNKNOWN, query the provider with a stable idempotency key where possible, and only then decide whether to retry, compensate, or ask for approval.

A minimal failure-injection test

You can test the dangerous race without a large distributed system:

Start worker A with lease token 1.
Pause A immediately before a side effect.
Let the lease expire.
Start worker B and assign token 2.
Let B perform the side effect.
Resume A and make it attempt the same side effect.
Verify that the side-effect store rejects token 1.
Verify that the run has one final ownership record and one reconciliation record.

Repeat the test with a delayed network response, a process restart, and a duplicate request. The expected result is not “the old worker stopped.” You cannot reliably guarantee that. The expected result is “the old worker could not mutate state after fencing.”

Useful evidence includes:

lease acquisition and renewal timestamps
owner and fencing token on every side effect
rejection count for stale tokens
time spent in OUTCOME_UNKNOWN
reconciliation decisions and their operator or policy source

Where hosting fits

An always-on agent runtime makes lease renewal and durable state easier to operate, but it does not create the safety property for you. If you run OpenClaw or another worker on managed infrastructure, keep the lease store and side-effect boundary explicit, test restart behavior, and verify what survives a rebuild.

For teams that do not want to maintain the base always-on host, managed OpenClaw hosting on Ampere is one option to evaluate. The important question is still architectural: can your worker prove current ownership, and can the downstream system reject stale ownership?

Run-lease checklist

Before calling an agent workflow failover-safe, verify:

ownership is stored durably per run
ownership changes issue a monotonically increasing fencing token
every irreversible step carries that token
the downstream side-effect boundary rejects stale tokens atomically
lease loss becomes an explicit state, not a generic retry
ambiguous tool outcomes enter reconciliation
restart and duplicate-request races are failure-injected regularly
rebuild documentation explains how leases, state, and credentials are restored

A timeout tells you that a worker may be gone. A lease plus a fencing token tells every other component whether that worker is still allowed to act. That distinction is what keeps a restart from becoming a duplicate side effect.

Your AI Agent Needs a Cost Circuit Breaker, Not a Monthly Budget

Zira — Fri, 14 Aug 2026 02:04:55 +0000

A monthly spend limit is useful for accounting. It is a poor safety control for an AI agent.

An agent can burn through a budget long before the invoice arrives: a retry loop, a tool that returns oversized context, a fallback model that is more expensive than the primary model, or several workers all repeating the same request. By the time someone notices the dashboard, the expensive behavior has already happened.

The control I want instead is a cost circuit breaker: a small state machine that reserves estimated spend before work starts, records actual usage, and changes what the agent is allowed to do as the budget is consumed.

This is not a claim that a circuit breaker makes model usage cheap. It makes overspend a bounded and observable failure mode.

1. Separate four kinds of budget

Do not put every limit into one 'max_tokens' field. Track at least these dimensions:

Run budget: the maximum estimated cost for one user request.
Tenant or project budget: the amount that a workload may reserve in a time window.
Retry budget: the number and cost of retries allowed for one operation.
Provider budget: a cap for each model/provider so fallback cannot silently consume the whole allowance.

A request can be under its token limit and still violate a provider or retry budget. The decision should be made against all four.

A minimal reservation record can look like this:

type BudgetReservation struct {
    RunID          string
    OperationID    string
    Provider       string
    EstimatedCents int64
    ActualCents    int64
    RetryIndex     int
    State          string // RESERVED, SETTLED, RELEASED, UNKNOWN
}

The important part is not the language. It is that the reservation has a stable OperationID and a durable state.

2. Reserve before dispatch, settle after the response

The dangerous order is:

Call the model.
Read the usage fields.
Decide whether the call was affordable.

That order has already spent the money. Instead:

Estimate the worst-case input and output cost.
Atomically reserve that amount against the applicable budgets.
Dispatch the request with the reservation ID.
Settle the reservation using provider usage data.
Release unused capacity, or mark the reservation UNKNOWN if the outcome is ambiguous.

The reserve operation needs to be atomic. Two workers must not both observe the same remaining balance and both receive permission.

A simple SQL shape is enough to make the invariant explicit:

UPDATE budget_windows
SET reserved_cents = reserved_cents + :estimate
WHERE budget_key = :key
  AND reserved_cents + settled_cents + :estimate <= limit_cents;

If the affected-row count is zero, reject or downgrade the operation. Do not enqueue it and hope a later worker notices.

3. Make the breaker change behavior

A useful breaker has more than open and closed states. For example:

NORMAL: full tool and model policy.
THROTTLED: lower concurrency and shorter output ceilings.
DEGRADED: allow read-only tools and a cheaper approved model.
OPEN: reject new work, but allow reconciliation of in-flight requests.
UNKNOWN: stop automatic retries until provider usage and delivery state are reconciled.

The transition should be based on reserved plus settled spend, not only settled spend. Otherwise a burst of concurrent requests can oversubscribe the remaining budget.

Example policy:

Remaining allowance	Policy
More than 40%	Normal execution
15% to 40%	Reduce concurrency and cap output
1% to 15%	Read-only or low-cost model only
0% or negative	Open for new work

The percentages are examples, not universal defaults. Tune them from your workload and record why a transition occurred.

4. Treat retries as new reservations

A retry is not free just because the original request failed from the application’s point of view. The provider may have processed it, and a timeout may leave the outcome unknown.

Give every attempt a stable operation identity plus an attempt number:

run_482 / summarize_invoice / attempt_2

Before retrying:

look up the provider request ID if one exists;
reconcile usage for the previous attempt;
check whether the tool side effect completed;
reserve the retry cost separately;
stop when the retry budget is exhausted.

This is especially important for always-on agents. A restart loop can turn one logical task into hundreds of billable attempts. If you run OpenClaw or another agent continuously, managed OpenClaw hosting on Ampere can be one hosting option to evaluate for the runtime, but hosting does not remove model-cost, retry, or credential risk. The breaker still belongs in the agent control plane.

5. Do not let fallback bypass the breaker

Fallback logic often has an accidental escape hatch:

def call_with_fallback(request):
    try:
        return call('cheap-model', request)
    except TimeoutError:
        return call('premium-model', request)

Both calls need the same budget checks. Better:

def dispatch(request, provider):
    estimate = price(provider, request.input_tokens, request.max_output_tokens)
    reservation = reserve_all(provider, estimate, request.run_id)
    if not reservation:
        raise BudgetOpen('no capacity for this provider')
    try:
        response = call(provider, request, reservation.id)
        settle(reservation.id, usage=response.usage)
        return response
    except TimeoutError:
        mark_unknown(reservation.id)
        raise

Fallback then becomes a policy decision made after reconciliation, not an exception handler that can spend outside the guardrail.

6. Test the failure paths deliberately

A cost control that only works on successful responses is not a control. Inject these cases in staging:

two workers reserve the final available cents concurrently;
the provider times out after accepting the request;
usage data arrives late or is missing;
a retry is attempted after the breaker opens;
a premium fallback is requested while the provider budget is exhausted;
a process crashes after reservation but before settlement;
a process crashes after settlement but before releasing the unused estimate;
the clock moves across a budget-window boundary;
a duplicate message is delivered to two workers.

For every case, assert an invariant: the ledger is reconcilable, no reservation is silently lost, no automatic retry bypasses the breaker, and the final charge is attributable to one run and attempt.

7. Measure the control, not just the invoice

Track:

estimated versus actual cost by provider;
reserved capacity that remains UNKNOWN;
retry cost per operation;
fallback rate and fallback cost;
time spent in each breaker state;
rejected work and its reason;
maximum concurrent reserved spend.

The useful question is not only “How much did this month cost?” It is “Which state transition prevented the next unbounded retry or fallback?”

A monthly budget tells finance what happened. A cost circuit breaker gives the runtime a chance to stop, degrade, or reconcile before a small failure becomes a large bill.

If you build agents, follow for practical control-plane patterns around state, permissions, recovery, and deployment rather than model demos alone.

Your MCP Server Needs a Capability Budget, Not Just Auth

Zira — Thu, 13 Aug 2026 22:02:56 +0000

Most MCP security checklists stop at “is this caller authenticated?” That is necessary, but it does not answer the operational question: what is this tool allowed to do during this run?

A useful boundary is a capability budget: a short-lived, explicit contract for each tool invocation. It should constrain the action, target, resource, quantity, and expiry. The model can request a tool, but the runtime decides whether the request fits the contract.

1. Define the budget next to the tool

Start with a deliberately boring schema:

type CapabilityBudget = {
  runId: string;
  tool: string;
  actions: string[];
  resources: string[];
  maxCalls: number;
  maxBytes?: number;
  expiresAt: string;
  approval: "none" | "human";
  policyVersion: string;
};

For example, a code-review run might receive:

{
  "runId": "run_8f2",
  "tool": "github.create_comment",
  "actions": ["comment"],
  "resources": ["repo:acme/api:pull:481"],
  "maxCalls": 1,
  "expiresAt": "2026-08-14T03:00:00Z",
  "approval": "human",
  "policyVersion": "p17"
}

The important part is what is absent: no repository-wide write permission, no issue creation, and no unbounded retry allowance.

2. Enforce it at dispatch time

Do not check the budget only when the agent plans the call. The queue, worker, and tool adapter should all treat the budget as untrusted input and revalidate it.

A dispatch decision can be reduced to:

Load the current run and budget.
Verify the budget is unexpired.
Verify the requested action and resource match exactly.
Atomically reserve one call and any byte quota.
Recheck the current policy and credential version.
Dispatch with the reservation ID.

If steps 4 and 5 cannot be made durable, a worker crash can turn one approved call into several attempts. That is a reliability bug as well as a security bug.

3. Separate planning from spending

Let the model propose a sequence, but make the runtime spend from the budget one reservation at a time. A plan such as “inspect, patch, test, notify” should not silently inherit write capability from the first step.

A minimal ledger makes this visible:

reservation	tool	requested	decision	outcome
r1	repo.read	1 call	allowed	confirmed
r2	repo.write	1 call	approval required	pending
r3	slack.send	1 call	denied	not dispatched

Keep pending and unknown distinct. Pending means no dispatch has been recorded. Unknown means dispatch may have happened but confirmation was lost. Only the latter requires provider lookup or an idempotency-key reconciliation before retry.

4. Test the failure modes

A capability budget is only real if it survives interruptions. Inject at least these cases:

The worker pauses after reservation but before dispatch.
The provider times out after accepting the request.
The policy changes while a job is queued.
The model changes the resource identifier between planning and dispatch.
Two workers race for the last allowed call.
A retry arrives after the budget expires.
A child tool asks for a broader capability than its parent run owns.

For every case, record the expected ledger state, whether a provider lookup is required, and whether a human must reapprove. A green test suite that never forces an unknown outcome is testing the happy path, not the boundary.

5. Make the hosting boundary explicit

If this runtime must stay available for queued work or browser-assisted tasks, hosting is part of the control plane. A managed runtime such as managed OpenClaw hosting on Ampere can be evaluated as one deployment option, but it does not replace capability checks, prompt-injection defenses, credential scoping, or reconciliation logic.

The questions to verify are practical: where durable run state lives, how workers restart, how credentials are mounted, how logs are retained, and how you rebuild the same policy version on a clean host. Treat the hosting choice as an availability and recovery decision, not as an authorization decision.

A compact acceptance checklist

Before calling an MCP integration production-ready, verify:

Every tool call has a run-bound, expiring capability budget.
Action and resource are matched at dispatch, not only during planning.
Call and byte limits are reserved atomically.
Policy and credential versions are rechecked before execution.
Parent and child tools cannot expand one another’s authority.
Timeouts produce unknown, not an automatic retry.
Provider IDs and idempotency keys support reconciliation.
Expiry, revocation, and clean-host restore are tested.

Authentication answers “who is asking?” A capability budget adds “what may happen, where, how often, and until when?” That is the boundary worth reviewing.

Your AI Agent Is Restarting in a Loop. Stop Treating It Like a Health Check

Zira — Thu, 13 Aug 2026 18:03:21 +0000

A process that is alive is not necessarily an agent that is healthy.

A supervisor can report “running” while the worker is crashing before it registers, restoring a corrupt state file, losing its credential lease, or replaying the same outbound action after every restart. Restarting harder does not fix those failure domains. It can make them worse by hiding the first useful error and multiplying side effects.

This article gives you a small restart-loop protocol: classify the failure, preserve the evidence, quarantine unsafe work, and only then decide whether to restart.

1. Separate process liveness from agent readiness

Use at least three states:

PROCESS_UP: the process accepts a local health request.
READY: configuration, durable state, credentials, and dependencies passed startup checks.
EXECUTING: the agent is allowed to claim new work.

Do not use a single /healthz endpoint for all three. A process can be up while its state database is locked or its credential lease has expired.

A useful readiness response should identify the failed domain without exposing secrets:

domain              status    evidence
runtime             ok        pid=1842, build=2026.08.13
state               fail      sqlite_busy_after=3s
credentials         unknown   lease_version=41
outbound_delivery   paused    pending=2, unknown=1
admission           blocked   reason=readiness_failed

The important part is the last line. A failed readiness check should stop new work, not necessarily kill the process.

2. Preserve the first failure, not just the last restart

Supervisors often produce a misleading story:

start worker
worker fails during recovery
supervisor restarts worker
logs rotate or the same error is buried
repeat

Record a restart episode before attempting another start. Give it an ID and attach:

boot ID and process ID
config hash and state-schema version
last durable run position
credential version, never the credential value
dependency readiness results
the first exception and its timestamp
whether a tool call or delivery was UNKNOWN at shutdown

A compact event model is enough:

auto_restart_episode(episode_id, boot_id, config_hash, state_position, readiness_failures, first_error, unknown_side_effects)

If the first error is not durable, you do not have a restart policy. You have a loop with a counter.

3. Add a restart budget and a quarantine state

A bounded restart budget prevents a broken worker from consuming all CPU, API quota, or queue capacity. For example:

MAX_RESTARTS = 5
WINDOW = 10 minutes
recent = episodes within WINDOW
if len(recent) >= MAX_RESTARTS: return QUARANTINED
else: delay = min(300, 2 ** len(recent))

The exact numbers are workload-specific. The invariant is not: after repeated failure, the worker must stop claiming new work and require a human or an automated rollback decision.

Quarantine should be explicit and observable:

no new admissions
leases expire or are returned safely
in-flight work becomes UNKNOWN unless its outcome is confirmed
outbound delivery retries pause
a diagnostic bundle is retained
recovery can start from a known-good configuration

Do not automatically delete the state directory as a “fix.” That may erase the only evidence needed to distinguish corrupted state from a bad deployment.

4. Test the four common restart-loop causes

Bad configuration

Deploy a config with an invalid tool or model endpoint. Verify that the worker enters NOT_READY, records the config hash, and does not claim work.

State incompatibility

Restore a state snapshot from an older schema. Verify that migration is explicit, reversible, and logged. A migration that partially writes before crashing needs a recovery position, not another blind restart.

Expired credentials

Revoke or expire the credential lease while the worker is stopped. On boot, it should report CREDENTIALS_NOT_READY, avoid tool calls, and request a fresh scoped lease.

Unknown side effect

Kill the process after dispatch but before the provider response. On restart, the run must remain UNKNOWN until the provider is queried or an operator makes a documented reconciliation decision. It must not blindly replay the action.

A useful failure matrix looks like this:

failure                  expected startup state    allowed action

bad config                NOT_READY                 diagnostic only

state migration error    QUARANTINED               rollback or repair

expired credential       NOT_READY                 lease renewal

unknown tool outcome     READY, admission paused    reconcile first

Make recovery evidence part of the deployment

Before putting an always-on agent on a VPS or managed runtime, prove that you can answer:

Which code and config produced this boot?
Which state was durable before the crash?
Which work was executing, queued, delivered, or UNKNOWN?
Which credentials survived, and what is their blast radius?
Can I rebuild the runtime without copying an opaque machine image?
Can I pause admissions without taking down diagnostics?

If you want a managed place to run an always-on OpenClaw or browser workload, managed OpenClaw hosting on Ampere is one option to evaluate. It does not remove prompt-injection, credential, or application-level recovery risk; you still need the state, admission, and reconciliation checks above.

The practical rule

A restart is an action, not a diagnosis.

First preserve the failure episode. Then classify process liveness, readiness, execution state, delivery state, and credential state separately. Quarantine after a bounded number of attempts. Reconcile UNKNOWN side effects before replaying anything.

That turns “the container keeps restarting” from a vague uptime problem into a testable recovery contract.

Your AI Agent Needs a Credential Lease, Not a Permanent API Key

Zira — Thu, 13 Aug 2026 14:17:58 +0000

Long-running agents turn a small credential mistake into a large incident.

A worker that can run overnight, spawn tools, or retry after a restart should not receive a permanent API key and keep it until the process dies. Treat access as a lease with an owner, scope, expiry, and revocation state.

The lease contract

Store these fields beside the run, not only in environment variables:

lease_id
run_id
principal
allowed_actions
allowed_resources
issued_at
expires_at
revoked_at
credential_version

A tool call is allowed only when all of these checks pass:

The lease exists and is unexpired.
The run is still authorized to use it.
The requested action and resource match the lease.
The credential version is current.
The worker has not entered a quarantine state.

The model can suggest a tool call. It must not decide whether the lease is valid.

Recheck at dispatch time

Checking access when a run starts is not enough. Queues, retries, and browser sessions create time gaps. Recheck immediately before dispatch:

def authorize(call, lease, policy, now):
    if lease.revoked_at is not None:
        return 'DENY_REVOKED'
    if now >= lease.expires_at:
        return 'DENY_EXPIRED'
    if lease.credential_version != policy.current_version(lease.principal):
        return 'DENY_STALE_VERSION'
    if call.action not in lease.allowed_actions:
        return 'DENY_ACTION'
    if call.resource not in lease.allowed_resources:
        return 'DENY_RESOURCE'
    return 'ALLOW'

Do not silently refresh a denied lease. Create a new authorization decision with a new lease ID, and record why the old one failed.

Make revocation observable

A useful event trail has one row per decision:

LEASE_ISSUED
CALL_PROPOSED
CALL_ALLOWED or CALL_DENIED
CREDENTIAL_PRESENTED
OUTCOME_CONFIRMED, OUTCOME_FAILED, or OUTCOME_UNKNOWN
LEASE_REVOKED

Never log the secret itself. Log the lease ID, credential version, provider request ID when available, and a hash of the resource identifier if the raw value is sensitive.

The failure test that matters

Run this test against a staging provider:

Start a run with a 10-minute lease.
Queue a tool call but pause the worker before dispatch.
Revoke the lease and rotate the credential.
Resume the worker.
Assert that dispatch is denied with DENY_REVOKED or DENY_STALE_VERSION.
Repeat with the provider accepting the request but timing out the response.
Mark the result UNKNOWN and reconcile using the provider request ID before retrying.

This catches the dangerous gap between the agent being allowed earlier and the agent being allowed now.

Hosting does not remove the boundary

If you run an always-on OpenClaw or browser worker on a managed runtime such as managed OpenClaw hosting on Ampere, the lease layer still belongs in your application. Hosting can simplify where a process runs; it does not decide which tenant, tool, resource, or credential version a worker may use.

A practical review checklist

Every lease has an expiry and explicit scope.
Dispatch rechecks policy after queueing and after retries.
Revocation reaches workers, browser sessions, and child tools.
Credential rotation invalidates old versions.
UNKNOWN outcomes are reconciled before retry.
Logs prove decisions without exposing secrets.
A staging test demonstrates that a paused worker cannot use a revoked lease.

The goal is not to make agents incapable of acting. It is to make every action attributable, bounded, revocable, and recoverable.

Your Agent Container Is Not Isolated Until You Test the Mounts

Zira — Thu, 13 Aug 2026 10:19:03 +0000

A container can make an AI agent easier to deploy without making it safe to operate.

The failure is usually not the image. It is the boundary around the image: bind mounts, Docker socket access, shared networks, environment variables, and persistent volumes. If an agent can reach the host filesystem or reuse a broad credential after a rebuild, running in Docker is only a packaging choice.

This article gives a repeatable mount-and-credential audit for OpenClaw-style agent runtimes and other tool-using workers.

Start with a threat model

Write down what the agent needs and what it must never reach.

Capability	Safer boundary
Read workspace files	Read-only mount, or a dedicated worktree
Write artifacts	One output directory, not the home directory
Call a browser	Separate browser service and disposable profile
Send messages	Narrow, revocable delivery credential
Control Docker	Never mount the Docker socket into the agent
Read host secrets	No host home, SSH, or cloud metadata mount

The useful question is not whether the container starts. It is what a prompt-injected tool call can reach when the model makes the wrong request.

Defaults that deserve an explicit review

A whole-project bind mount when the directory contains secrets or other repositories.
/var/run/docker.sock mounted into the agent.
host networking, which removes useful network separation.
Passing every host environment variable through an env file.
Reusing a long-lived browser profile across unrelated tasks.
One opaque writable volume for both durable state and disposable cache.

These patterns are not automatically wrong in every development environment. They are risky because they make the effective permission set larger than the task.

A smaller Compose baseline

Start with a non-root UID, a read-only root filesystem, dropped capabilities, and a temporary directory for scratch space:

services:
  agent:
    image: example/agent:latest
    user: "10001:10001"
    read_only: true
    cap_drop: ["ALL"]
    security_opt:
      - no-new-privileges:true
    tmpfs:
      - /tmp:rw,noexec,nosuid,size=256m
    volumes:
      - ./agent-workspace:/workspace:rw
      - agent-state:/var/lib/agent:rw
    environment:
      AGENT_WORKSPACE: /workspace
      DELIVERY_TOKEN_FILE: /run/secrets/delivery_token
    secrets:
      - delivery_token

  secrets:
    delivery_token:
      file: ./secrets/delivery_token

  volumes:
    agent-state:

This is not a complete security policy. It is a boundary that can be inspected. Add outbound access through a small egress proxy or separate delivery worker when the agent does not need general network access.

For an OpenClaw deployment that must stay available, a managed runtime such as managed OpenClaw hosting on Ampere can reduce the amount of host maintenance you own. It does not remove the need to review mounts, credentials, browser profiles, or recovery behavior.

Test the boundary from inside the container

Do not stop at reading YAML. Run tests against the running process and record the result.

1. Check identity and capabilities

docker compose exec agent id
docker compose exec agent sh -c 'grep CapEff /proc/1/status'
docker inspect agent --format '{{json .HostConfig.CapDrop}}'

The process should not run as root and should not retain unnecessary Linux capabilities.

2. Enumerate mounts

docker inspect agent --format '{{range .Mounts}}{{println .Source "->" .Destination "rw=" .RW}}{{end}}'

Fail the deployment if a source is broader than the task requires, if a sensitive host path appears, or if a cache is accidentally writable.

3. Probe for host-control paths

docker compose exec agent sh -c 'test ! -S /var/run/docker.sock && test ! -d /root/.ssh && test ! -f /proc/1/root/etc/shadow'

A failed probe is not proof of compromise. It is proof that the boundary needs review.

4. Test credential replacement

Create a short-lived test credential with a unique identifier. Revoke it, restart the agent, and confirm that the old credential is rejected, the agent cannot read the host shell environment to find a replacement, logs record the denial without printing the secret, and a clean rebuild does not silently restore the revoked credential.

A credential file should have an owner, scope, expiry, and revocation procedure. Being in a secret store is not a procedure.

Persistence needs two inventories

Rebuildable: image layers, package caches, temporary browser data, exported logs, and generated files that can be recreated.

Durable: agent identity, pending work, idempotency keys, policy version, provider cursors, approved destinations, and encrypted credential references.

If both categories share one opaque volume, operators restore too much or too little. A clean-room restore should prove that durable state returns while stale browser cookies, abandoned locks, and revoked tokens do not.

Failure-injection checklist

Run this before calling the deployment recoverable:

Delete the container while a tool call is running.
Remove the workspace mount and verify the agent fails closed.
Revoke the delivery credential during a retry.
Restore the state volume into a new container with a different image digest.
Start two workers against the same state volume.
Interrupt the process after it records intent but before the external side effect.
Interrupt it after the provider accepts the request but before the response is stored.

For every case, record confirmed, not started, or unknown. Unknown is a first-class state. Retrying it blindly is how a recovery script sends duplicate messages or repeats a destructive tool call.

A deployment gate for CI

A lightweight gate can catch regressions:

no Docker socket mount;
no host network mode;
no sensitive host-path mounts;
non-root user;
dropped capabilities;
explicit writable mounts;
secrets passed by file or workload identity, not broad environment inheritance;
state and cache volumes named separately;
image digest recorded;
restore and revocation tests attached to the release.

The goal is not to claim perfect isolation. It is to make the effective boundary visible, narrow, and testable.

If you build or host AI agents, follow for practical failure tests around state, identity, delivery, and recovery. The model is only one component. The container boundary decides what happens when the model, tool, or operator is wrong.