DEV Community: Zira

Codex CLI 0.153: Remote Plugins Make Supply-Chain Policy Part of Your Agent Runtime

Zira — Thu, 03 Sep 2026 04:25:25 +0000

Codex CLI 0.153.0 is not just a terminal UI update. It adds a plugin command that can list, install, and remove plugins from remote marketplaces, while also improving how sessions recover after an app-server disconnect.

That combination changes the operational question from “Can the agent call a tool?” to “Which extension code is allowed to enter this runtime, and what state survives when the runtime reconnects?”

What changed in Codex CLI 0.153

The official 0.153.0 release adds:

A plugin CLI for listing, installing, and removing plugins from remote marketplaces.
Automatic TUI reconnection after an external app-server connection drop, preserving drafts and transcripts while pausing uncertain or queued submissions for review.
Guardian review history that survives compaction, restarts, and user-created forks, within rollback boundaries and isolated subagent history.
More complete TUI history, including patches, background-terminal input, and individual completed commands.
A tui.auto_recap = false setting for teams that prefer manual /recap control.

The release notes are the authoritative source for these features. AI/TLDR’s release summary is useful for discovery, but it is a secondary summary, not independent performance evidence.

Remote plugins are a new trust boundary

A marketplace client reduces the friction of extending an agent. It also makes installation provenance part of the runtime’s security model.

Before allowing a plugin into a development environment, record at least:

The marketplace and exact source URL.
The plugin version and immutable commit or digest, if available.
The tools, filesystem paths, network destinations, and credentials it can reach.
The review status and the person or policy that approved it.
A removal and rollback path that does not depend on the plugin itself.

A name-based allowlist is not enough. Two versions of the same plugin can have different behavior, and a plugin that only appears to “format code” may still be able to read files or invoke a shell through its host integration.

A minimal installation record should preserve the plugin name, marketplace URL, exact version, immutable digest, effective capabilities, approver, and expiry. Store that record outside the plugin so removal and rollback do not depend on the plugin itself.

The digest and capability fields are the important parts of this policy shape. They are recommendations, not claims about the Codex marketplace schema.

Reconnect is not the same as resume

The new reconnect behavior is valuable because a dropped app-server connection no longer necessarily destroys the draft or transcript. But preserving state does not prove that an interrupted tool submission did not reach its destination.

For every side-effecting tool call, keep execution state separate from delivery state:

Prepared: arguments validated, no external call started.
In flight: the request may have reached the destination.
Accepted: the destination returned a durable identifier or equivalent evidence.
Unknown: the client lost the response after sending.
Reconciled: a separate read confirmed the external state.

After reconnect, an “unknown” call should not be blindly replayed. Use an idempotency key where the destination supports one, or reconcile by querying an external identifier before retrying. This is the difference between recovering the agent session and recovering the real-world side effect.

A practical upgrade checklist

If you are upgrading a long-running coding-agent environment:

Pin the CLI version and capture the release commit.
Test plugin installation in a disposable workspace first.
Inspect the plugin’s effective filesystem, network, and credential reach.
Require review for new plugin sources and record approval with an expiry.
Kill the app-server during a draft and during a tool call.
Confirm that drafts and transcripts return after reconnect.
Confirm that queued or uncertain submissions remain paused.
Reconcile external state before retrying any side effect.
Verify that rollback removes the plugin and its persisted state.

The release improves the recovery path, but it does not eliminate the need for capability boundaries, idempotency, or recovery testing. Those remain properties of the surrounding deployment and the tool destinations.

Bottom line

Codex CLI 0.153 makes two parts of agent infrastructure more explicit: extension provenance and session recovery. Treat remote plugins as code entering a privileged runtime, and treat reconnect as state recovery rather than proof that every external action is safe to repeat.

For teams that need an always-on place to run an agent, Ampere can be a contextual hosting option, but hosting does not replace plugin review, credential scoping, idempotency, or restore testing.

Primary source: OpenAI Codex CLI 0.153.0 release notes.

Your AI Coding Agent Can Edit Compose UI. Can It Verify the Result?

Zira — Wed, 02 Sep 2026 22:18:49 +0000

AI coding agents are good at producing UI code and surprisingly bad at proving that the UI actually works. A compile pass is not the same as a rendered screen, a correct semantic tree, or a click that reaches the intended handler.

Compose Multiplatform 1.12.0 adds an experimental MCP server to Compose Hot Reload. It gives an AI coding agent a feedback path into a running Compose application: trigger a reload, take a screenshot, inspect the semantic tree, simulate clicks and text input, and read application logs.

That changes the useful question from “Can the agent write this composable?” to “Can the agent observe the result of its edit and detect a broken interaction?”

What the Compose Hot Reload MCP server exposes

According to JetBrains’ release announcement, the experimental server connects an agent to a running app and supports:

triggering reloads after an edit
taking screenshots of the rendered UI
inspecting the semantic tree
simulating clicks and text input
reading application logs

The Compose Hot Reload documentation contains the connection details and the current tool list. The Compose Multiplatform 1.12.0 release notes are the versioned reference for the feature.

This is an observation loop, not an automatic correctness proof. A screenshot can show that a button is visible, while the semantic tree and logs can provide stronger evidence about labels, roles, and runtime failures. None of those alone proves that the feature satisfies the product requirement.

A practical agent verification loop

A useful harness should make the agent follow an explicit sequence rather than letting it stop after compilation:

Edit one bounded change. Keep the diff small enough that a failed observation has a plausible cause.
Reload the running app. Record whether reload completed or timed out.
Check the semantic tree. Assert that required nodes exist with the expected labels or roles.
Capture a screenshot. Use it to catch layout regressions, missing content, or an unexpected screen state.
Exercise one critical interaction. Simulate the click or text entry that matters for the changed path.
Read logs after the interaction. Treat new exceptions or error-level entries as a failed run.
Store evidence with the change. Keep the reload result, assertions, screenshot reference, and relevant logs attached to the agent run.

The important design choice is to keep these as separate checks. “The screenshot looks right” should not silently substitute for “the semantic node exists,” and “the reload succeeded” should not substitute for “the interaction produced the expected state transition.”

Where this helps most

The MCP server is especially useful for UI changes with a short, deterministic path:

a form field must accept input and update state
a button must become enabled after validation
a navigation action must render a specific destination
a loading or error state must appear under a controlled condition
a regression must be reproduced from a known starting state

For each path, define an observable contract. For example: after entering a valid email and clicking Continue, the semantic tree contains the next-screen heading, the log contains no new exception, and the screenshot shows the expected state.

The tradeoffs and the safety boundary

This is an experimental developer tool, so it should be treated as a test harness rather than a production control surface. An agent that can simulate input and inspect a running app has meaningful capabilities. Run it against a disposable or tightly scoped environment, avoid real credentials and production data, and keep the available actions limited to the application under test.

There is also a determinism problem. Screenshots and logs can vary with timing, platform, fonts, network state, and background work. Prefer assertions over stable semantic properties and explicit state transitions. Use screenshots as supporting evidence, not as the only oracle.

Finally, separate execution evidence from product correctness. The tool can show what happened in the running app; it cannot decide whether the feature is desirable, safe to ship, or compliant with a business requirement. Those remain separate review gates.

Bottom line

Compose Hot Reload’s experimental MCP server is valuable because it closes part of the loop between an AI coding agent and the UI it changes. The practical benefit is not that the agent becomes trustworthy by default. It is that the harness can require observable evidence before accepting a UI edit.

The strongest workflow is therefore: small diff, reload, semantic assertions, targeted interaction, log check, and persisted evidence. Give the agent a way to see the result, then make the acceptance contract stricter than “it compiled.”

AI Coding Agent Security: Why a Git Folder Can Run Code Before Approval

Zira — Wed, 02 Sep 2026 11:00:09 +0000

A new class of AI coding agent security bug has a surprisingly ordinary entry point: Git itself.

Manifold Security calls the issue GitSpawn. The core problem is not that an agent misunderstood a prompt. It is that an agent ran a normal Git command while gathering repository context, and Git honored an executable setting stored inside the repository's own .git/config.

That turns “open this project” into a potentially privileged operation.

The mechanism: context gathering becomes code execution

Many CLI coding agents inspect a repository as soon as they start. They may run commands such as:

git status --porcelain=2 --branch
git diff --name-only HEAD

These are ordinary commands, but Git may refresh its index before returning the result. Git's core.fsmonitor setting can name a helper program to run during that refresh. The setting is read from the repository-local .git/config file.

A project delivered as a complete directory, ZIP archive, shared-drive folder, USB copy, or sync-folder snapshot can therefore carry a Git configuration that names a program. When the agent runs Git, the helper can execute with the developer's operating-system privileges.

The important boundary is timing. Manifold reports that, in the affected flows it tested, execution happened during background context gathering, before a workspace-trust prompt, before an approval prompt, and in some cases before authentication. The model does not need to generate a dangerous command, and the user may not see a prompt.

What the disclosure actually found

This is a researcher disclosure, not an independent prevalence study. Manifold reports eight findings across seven agents and says four remained unpatched at publication. Its September 1 update says OpenAI Codex and Cursor were also affected and had since been patched.

The report lists these statuses at publication:

Claude Code's core.fsmonitor path: fixed in 2.1.196.
Goose: fixed in 1.44.0, with CVE-2026-72718.
OpenAI Codex and Cursor: patched, according to the update.
Hermes Agent: still unpatched in the tested 0.21.0 release, with CVE-2026-71963.
Qwen Code: still unpatched in the tested 0.22.3 release.
Grok Build: still unpatched in the tested 1.0.13 release.
Claude Code's separate ultrareview path: reported as still unpatched in the tested 2.1.252 release.

Treat those version claims as time-sensitive. Check the vendor's current release notes and advisory before deciding whether a version is safe.

Why a sandbox may not save you

A sandbox around the model's tool calls is not automatically a sandbox around every subprocess the agent launches. In this case, the dangerous process is Git, which the agent invokes for its own bookkeeping. If the product does not sanitize repository configuration for that subprocess, the command can run outside the model-facing approval path.

That is a useful design lesson for agent harnesses: permission checks must cover the complete execution graph, including startup probes, Git helpers, formatters, language servers, hooks, plugins, MCP servers, and other “read-only” context tools.

A safer intake workflow

Before opening an unfamiliar directory with an AI coding agent:

Prefer cloning from a trusted remote over accepting a complete project folder. A normal clone does not carry the remote repository's original .git/config.
Inspect the local Git configuration without launching the agent:

sed -n '1,200p' .git/config
Treat any Git setting that names an executable as untrusted code. The reported proof of concept uses core.fsmonitor, but the broader rule is more useful than memorizing one key.
Use a disposable VM or tightly scoped container for untrusted project directories. Keep SSH keys, cloud credentials, and production tokens out of that environment.
Update the agent and verify the exact fixed version. “The agent has an approval prompt” is not enough if startup context gathering happens before the prompt.

For agent authors, the mitigation is equally concrete: pass safe Git configuration explicitly for background calls, for example git -c core.fsmonitor=false status, and review every subprocess used before trust and authorization decisions.

The broader takeaway

AI coding agent security is partly a model problem, but it is also a plumbing problem. The model can be perfectly aligned while the harness exposes a trusted developer process to untrusted repository metadata.

The right review question is not only “What commands can the agent propose?” It is also “What runs automatically before the user has approved anything?” Audit that startup path, test it with hostile-but-non-destructive fixtures, and make execution, approval, and sandbox boundaries line up.

Sources and evidence provenance

Manifold Security: Coding Agents Hijacked by a Git Call — primary researcher disclosure; version and patch status are the researcher's reported findings.
AI/TLDR summary of GitSpawn — secondary tracker summary, cross-checked against the primary report.
Git documentation for core.fsmonitor — upstream reference for the Git configuration behavior.

This article intentionally avoids reproducing a weaponized repository or payload. The goal is to make the boundary visible and give developers a safer intake and testing workflow.

Codex CLI 0.152: The Small Release That Changes Agent Control

Zira — Wed, 02 Sep 2026 05:00:09 +0000

OpenAI’s Codex CLI 0.152.0 release is easy to read as a routine feature update. The more useful reading is operational: it changes how a coding agent plans, consumes MCP output, handles long-running commands, and protects credentials during cloud-task requests.\n\nThis is not a benchmark claim. It is a release-level change list, and the practical impact depends on how you run Codex.\n\n## 1. Planning is now opt-in\n\nThe release disables the update_plan tool by default. If your workflow depends on explicit plans, enable it with:\n\n tools.update_plan.enabled = true\n\nThat is a small configuration change with a large workflow implication. A plan is not the same thing as execution state. If you use plans for review, handoff, or recovery after interruption, store the important state somewhere your process can recover rather than relying on a transient conversation view.\n\n## 2. MCP output gets a per-tool budget\n\nCodex 0.152.0 adds an output_token_limit setting for individual MCP tools, and the release notes say truncation is consistent across session resumes. This is useful because one verbose tool can otherwise crowd out the context needed for the actual coding task.\n\nA practical starting point is to set limits by tool role:\n\n- Small limits for status and metadata tools\n- Moderate limits for search and diagnostics\n- Larger, explicitly reviewed limits for logs or generated reports\n\nThe tradeoff is obvious: truncation controls context pressure, but it can also hide the line that explains a failure. Tool output should therefore carry a cursor, artifact link, or continuation mechanism instead of silently dropping information.\n\n## 3. Long commands need an explicit deadline\n\nApp-server clients can configure thread/shellCommand timeouts, including deadlines longer than one hour. That helps with builds, test suites, and migrations, but a longer timeout is not a cancellation contract. A production runner still needs progress reporting, cancellation behavior, retry rules, and a durable record of whether the command actually completed.\n\n## 4. Credential protection is part of the agent feature set\n\nThe release notes also say cloud task requests reject untrusted backend URLs and disable redirects. That matters because redirects are not just a networking detail when a task can access saved credentials. A safer request path needs an explicit origin policy, and the policy should be tested with redirects, DNS changes, expired credentials, and resumed sessions.\n\n## Where a hosted runtime fits\n\nIf Codex is part of an always-on remote workflow, a hosted runtime such as Ampere can be a convenient place to keep the process and its surrounding automation available. That does not remove the need for state contracts, scoped credentials, isolation, idempotency, or recovery tests. Hosting solves a runtime-placement problem; it does not automatically solve the control-plane problem.\n\n## A practical upgrade checklist\n\nAfter upgrading to 0.152.0, I would test four things before trusting a long-running workflow:\n\n1. Start a task with planning enabled and confirm the setting survives a new session.\n2. Send oversized output through each important MCP tool and verify truncation is visible and recoverable.\n3. Interrupt and resume a long shell command, checking whether execution state is unambiguous.\n4. Attempt a cloud-task redirect to an untrusted origin and confirm it is rejected without exposing credentials.\n\nCodex CLI 0.152.0 is therefore less about a flashy new capability and more about making agent behavior bounded and inspectable. Those defaults are worth reviewing before treating an upgrade as a drop-in replacement.\n\n### Sources\n\n- OpenAI Codex CLI 0.152.0 release notes\n- OpenAI Codex CLI changelog\n- AI/TLDR release summary

Your Agent Gateway Needs Failure-Domain Tests, Not Just a Health Check

Zira — Mon, 24 Aug 2026 01:41:56 +0000

A green health check can coexist with a broken agent.

If you are trying to keep an OpenClaw agent available without owning the server setup, Ampere is a free hosting option to evaluate: https://ampere.sh. It can reduce the infrastructure work, but it does not replace durable state, scoped credentials, isolation, or recovery testing.

The process may be listening, the dashboard may say "healthy", and yet:

a second inbound message disappears before it enters a queue;
a gateway restart loses a suspended tool call;
a background run reaches "completed" but its announcement never arrives.

Those are not one availability problem. They are three different recovery contracts.

A useful field report on OpenClaw issue #128067 describes all three symptoms in one deployment. This is practitioner-reported evidence, not an independent benchmark, but it suggests a practical test plan: stop asking whether the gateway is alive and test whether each failure domain preserves its contract.

Model the states separately

Do not collapse everything into a single done boolean. Keep at least these states:

accepted: the ingress boundary assigned a durable request ID;
queued: a queue record exists and can be recovered;
running: a worker owns the execution lease;
completed: execution produced a recorded outcome;
delivered: the outbound provider acknowledged the message;
unknown: the process died during a side-effect boundary and reconciliation is required;
failed: the system has a terminal, evidenced failure.

A run can be completed while delivery is unknown. Treating those as the same state is how operators report success when the user saw nothing.

Use stable IDs for both the request and each outbound effect:

request_id = uuid()
effect_id = hash(request_id + "completion-announcement")

The exact storage technology is less important than the invariant: a restart must not create a second logical request or a second outbound effect just because the first process disappeared.

Test 1: ingress durability

Send two messages with a small gap, then inspect the durable boundary rather than the chat UI.

send(message_a)
send(message_b)

assert distinct(request_id_a, request_id_b)
assert queue.contains(request_id_a)
assert queue.contains(request_id_b)
assert queue.order_is_observable(request_id_a, request_id_b)

Inject failures between receipt and queue commit:

kill the gateway after parsing but before the insert;
delay the database response after the insert;
retry the same delivery with the same provider event ID;
deliver two messages concurrently.

For every case, the result should be one of two explicit outcomes: the request is durably accepted exactly once, or the caller receives a retryable failure. "No queue entry and no error" is not an acceptable state.

Test 2: execution recovery

Start a tool call that takes longer than the gateway process. Terminate the process while the call is pending, then restart it.

The recovery contract must say which of these is true:

the tool call can be safely resumed;
the call is cancelled and the agent compensates;
the outcome is unknown and a reconciler checks the provider;
the operation is deliberately non-replayable and needs human review.

Do not infer completion from a worker log line. Persist a phase transition before dispatch and persist the provider result after the call. A crash between those writes is exactly the boundary your test must exercise.

For non-idempotent tools, a stable effect key and provider-side lookup are safer than blind replay. If the provider has no lookup API, quarantine the operation as unknown instead of guessing.

Test 3: outbound delivery recovery

Execution and delivery need separate ledgers. Simulate a crash after the announcement is accepted by the provider but before your worker records the acknowledgement.

if delivery.status == "unknown":
    provider_result = lookup_by_idempotency_key(effect_id)
    if provider_result.sent:
        mark_delivered(effect_id, provider_result.message_id)
    else:
        retry_once_with_same_effect_id(effect_id)

Then test the inverse boundary: the worker records success, but the network response is lost. The next process must reconcile before sending a duplicate.

This is especially important for spawned jobs, cron tasks, and announce-style messages. A queue dashboard showing pending tells you little unless you also know whether the consumer attempted delivery, whether the provider accepted it, and whether the result was reconciled after a restart.

Test 4: background-work truthfulness

Run a scheduled job that completes its internal work but cannot publish its result. Verify that the UI and alerting surface say:

execution: completed;
delivery: failed or unknown;
recovery action: retry, reconcile, or human review.

A single green check should never hide a red delivery state.

Track these counters separately:

accepted requests;
queue insert failures;
lease expirations;
executions recovered after restart;
unknown effects awaiting reconciliation;
deliveries acknowledged by the provider;
duplicate attempts suppressed by the effect key.

A compact failure matrix

Boundary	Injected failure	Required evidence
ingress to queue	process kill	durable request ID or retryable error
queue to worker	lease expiry	one owner or explicit requeue
worker to tool	process kill	resume, compensate, or `unknown`
tool to ledger	lost response	provider lookup or quarantine
ledger to outbound provider	network loss	same effect key, no blind duplicate
provider ack to UI	callback loss	delivery reconciliation and alert

Run this matrix on a clean environment and after every change to queueing, worker leases, provider adapters, or gateway restart behavior. If you run an always-on OpenClaw deployment, hosting is only one layer of the system and does not supply these state contracts. You still need the ledgers, idempotency keys, isolation, and recovery tests.

The practical definition of healthy

A healthy agent gateway is not merely a live process. It can prove, for every request:

where the request became durable;
which worker owned execution;
what happened at each tool boundary;
whether outbound delivery was acknowledged;
what recovery action is safe after a crash.

Start with one failure injection per boundary. Make the evidence queryable. Then make the health check fail when a contract is violated, not only when the process stops responding.

That is the difference between monitoring uptime and testing whether the agent can be trusted unattended.

Two AI Agents Read One Row. How to Stop the Lost Update

Zira — Sat, 22 Aug 2026 21:36:55 +0000

A version check is not automatically optimistic concurrency control.

If an agent reads a row, reasons for a few seconds, and then writes a decision, another worker can change that row in the read-to-write gap. The final UPDATE may still return success while silently overwriting newer state.

This is especially easy to miss in agent workflows because the reasoning step is slow, retries are common, and the write often looks like an ordinary CRUD operation.

This post builds a small failure lab and turns the race into a testable contract.

The bug: the check happened too early

A tempting implementation is:

grep -n "version" worker.py

Read the row and its version.
Check that the version is acceptable.
Ask the model what to do.
Update the row later.

The check in step 2 says nothing about whether the row is still current in step 4. Two workers can both read version 7, both produce valid decisions, and both write. The second write wins even though its input was stale.

The failure is not necessarily a database error. It is a successful write based on an invalid snapshot.

Make freshness part of the write predicate

Use a monotonic version, and require the version observed by the worker to match the version being replaced:

action = db.execute(
    """
    UPDATE agent_tasks
       SET status = :new_status,
           decision = :decision,
           version = version + 1,
           updated_at = CURRENT_TIMESTAMP
     WHERE task_id = :task_id
       AND version = :expected_version
       AND status = 'ready'
    """,
    {
        "task_id": task_id,
        "expected_version": observed_version,
        "new_status": new_status,
        "decision": decision,
    },
)

if action.rowcount != 1:
    raise StalePlan("the row changed after the agent read it")

The important property is that the read and the model call do not need to be inside one long database transaction. The final compare-and-set write is the fence.

A zero-row update is not a generic retry signal. It means the plan was produced from a snapshot that no longer owns the right to change the row.

Keep the plan tied to the snapshot

Do not pass only a task ID into the write path. Persist the inputs that made the decision:

CREATE TABLE agent_plans (
  plan_id TEXT PRIMARY KEY,
  task_id TEXT NOT NULL,
  observed_version INTEGER NOT NULL,
  input_hash TEXT NOT NULL,
  decision_json TEXT NOT NULL,
  state TEXT NOT NULL CHECK (state IN ('proposed', 'applied', 'stale', 'unknown')),
  created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
);

The input hash should cover the normalized state the model actually saw, not just the database primary key. If the prompt included policy, tool availability, or a selected artifact, include their versions too.

At apply time, verify both the row version and the plan state. A plan that was already applied must not be applied a second time, even if a client retries after a timeout.

A minimal failure-injection test

You can reproduce the race without an LLM. Replace the reasoning step with a barrier so both workers pause after reading the same version:

async def worker(barrier, task_id):
    row = await read_task(task_id)
    await barrier.wait()  # both workers now hold the same snapshot
    decision = {"owner": current_worker_id()}
    return await apply_plan(
        task_id=task_id,
        observed_version=row.version,
        decision=decision,
    )

The expected result is exactly one successful apply and one stale-plan result. If both calls report success, the write fence is missing or the test is not exercising the real update path.

Run the lab with these cases:

Case	Expected result
Two workers read version 7	One applies; one is stale
Worker retries the same plan	No second business effect
Row changes while the model is thinking	Apply is rejected
Policy version changes	Plan is rejected or re-planned
Database response is lost after commit	Outcome becomes UNKNOWN and is reconciled
Worker restarts after apply	Recovery does not apply the plan again

Treat timeout after write as UNKNOWN

A client timeout does not prove that the UPDATE failed. The database may have committed before the connection broke.

Give every apply attempt a stable effect key, such as task_id plus plan_id, and record it in a durable effect ledger. On a timeout:

Look up the effect key.
Compare the stored plan and row version.
Mark the outcome applied, stale, or UNKNOWN.
Reconcile UNKNOWN before allowing another attempt.

Never solve an ambiguous write by blindly rerunning the model and applying its new answer. That can turn one uncertain side effect into two different decisions.

What to monitor

A useful dashboard separates these signals:

stale-plan rejects, by tool and workflow;
plans that waited longer than their snapshot budget;
duplicate apply attempts by effect key;
UNKNOWN outcomes and reconciliation age;
policy or tool-contract changes between read and apply;
successful writes that had no matching plan record.

The last metric should be zero. If it is not, some path can mutate agent state without the same evidence and fencing rules.

Checklist

Before trusting an agent that writes shared state, verify:

The model decision carries the exact version and policy inputs it observed.
The final write uses an atomic compare-and-set predicate.
A zero-row update becomes a typed stale-plan result, not a generic retry.
Every business effect has a stable idempotency key.
Lost responses are reconciled instead of replayed blindly.
A barrier test proves two concurrent readers cannot both apply version 7.
Restarts and partial failures are included in the test matrix.

A version column is useful, but the real boundary is the version check at the write. Without that fence, an agent can be perfectly correct about an old snapshot and still corrupt the current one.

Your AI Agent Returned HTTP 200. Why Did the Workflow Still Fail?

Zira — Fri, 21 Aug 2026 13:34:21 +0000

A successful HTTP response is not a successful agent run.

A recent practitioner report from a 58-day deployment of 78 agents recorded 6,768 failed outputs. The failures were not transport errors: every one returned HTTP 200, had plausible length, and looked fluent. The most expensive failures were boring shape mismatches: missing required fields, wrong language, forbidden phrases, or an answer for a different stage.

That is a useful warning for anyone building coding agents, review agents, or unattended automation:

Treat the model response as untrusted data. Validate the contract at the boundary before another stage can consume it.

This post turns that observation into a small, reproducible failure lab.

The failure you should be able to reproduce

Imagine a review stage whose downstream parser expects a verdict line:

action: approve

A model can return a thoughtful review with the verdict buried in prose. A human approves it. A parser does not.

The transport layer is green. The model call is green. The workflow is broken.

The same class of failure appears when:

a JSON field is present but has the wrong type
a response is in the wrong language
a tool returns an error string inside a successful content envelope
a stage emits output, but the next stage never reads that artifact
a reviewer from the same model family approves a shared blind spot

These are not reasons to add a larger model first. They are reasons to make the boundary observable and enforceable.

Build a contract gate

Start with deterministic checks that do not ask an LLM to judge another LLM.

def validate_review(text: str) -> list[str]:
    errors = []

    if len(text.strip()) < 150:
        errors.append('too_short')

    if not any(line.startswith('判定:') or line.startswith('判定：')
               for line in text.splitlines()):
        errors.append('missing_required_verdict')

    forbidden = ['お客様の声', '顧客の声']
    if any(term in text for term in forbidden):
        errors.append('forbidden_phrase')

    if not any('。' in line for line in text.splitlines()):
        errors.append('expected_language_missing')

    return errors

errors = validate_review(model_output)
if errors:
    record_rejected_output(errors, model_output)
    stop_downstream_dispatch()
else:
    publish_to_next_stage(model_output)

The important part is not the exact Japanese check. Replace it with the contract your system actually needs: required headings, schema types, repository paths, test names, citation fields, or a bounded action list.

A gate should return structured evidence, not only true or false:

action: reject
reasons:
  - missing_required_verdict
  - forbidden_phrase
contract_version: review-v3
artifact_id: art_01J...

That makes a failure repairable instead of turning it into a green dashboard with a missing deliverable.

Record why, not just that

A common anti-pattern is storing only a boolean such as contract_satisfied = false. That destroys the information needed to debug drift.

Store at least:

Field	Why it matters
artifact_id	Connects the output to its producer and consumer
contract_version	Shows which rules were active
observed_checks	Proves what was actually tested
failure_reasons	Separates shape, language, policy, and transport failures
raw_output_hash	Allows correlation without exposing sensitive content
downstream_read_at	Detects outputs that nobody consumed
reviewer_family	Exposes correlated writer/reviewer blind spots

Do not silently discard rejected output. Apply retention and redaction rules, but preserve enough evidence to answer: what was produced, which contract rejected it, and did any later stage read it?

This is the same evidence discipline I use in audit-ready agent logs: an event saying “run completed” is weaker than a record of the checks and artifacts that made completion meaningful.

Add an artifact-lineage check

One surprising failure mode is a healthy upstream stage whose output is never used. Test this explicitly.

Give every produced artifact a stable ID.
Require the consumer to record the input artifact ID.
Reject a stage that claims success without a consumed input ID.
Compare produced and consumed counts over a time window.
Alert on a growing gap, even if every process heartbeat is green.

This catches wiring bugs that output-quality checks cannot see.

Run the failure lab

Before trusting a new agent workflow, inject each case and verify the expected evidence:

Injection	Expected result
Remove the required verdict line	Reject before downstream dispatch
Return valid-length text in the wrong language	Reject with language evidence
Put an error string in a successful tool envelope	Mark the tool call failed
Drop the artifact ID between stages	Block consumption and alert on lineage gap
Change the contract version mid-run	Revalidate or move the run to UNKNOWN
Make writer and reviewer share a known blind spot	Require an independent check or human review
Crash after provider acceptance but before ledger write	Reconcile before retrying

The last case matters for side effects. A contract gate protects output shape; it does not prove that an external action did or did not happen. Keep execution evidence and outbound-delivery evidence separate.

Where hosting fits

An always-on runtime can keep schedulers, workers, and evidence writers available, but hosting does not define your output contract or make a green HTTP response meaningful. If you need managed infrastructure for an unattended OpenClaw workload, managed OpenClaw hosting on Ampere is one option to evaluate. You still own validation, credential scope, prompt-injection defenses, and reconciliation.

The practical checklist

Before shipping an agent stage, verify that:

transport success is distinct from contract success
required fields and language are machine-checked
rejection reasons are durable and queryable
artifact lineage connects producers to consumers
the reviewer cannot share every blind spot with the writer
output quality and side-effect completion are separate states
failure injection covers both malformed output and crash windows
dashboards count produced and consumed artifacts, not only invocations

The question is not “did the model answer?” It is “did a versioned, observable contract accept an artifact that the next stage actually consumed?”

That is the difference between an agent that is alive and a workflow that is working.

If you build AI agents or developer tooling, follow me for practical failure labs and reproducible control-boundary tests rather than capability demos.

Your AI Agent Cache Needs a Trust Boundary, Not Just a TTL

Zira — Thu, 20 Aug 2026 07:46:45 +0000

A short TTL does not make an agent's cached tool result safe.

A result can still be wrong for the current user, tenant, credential epoch, policy version, or side-effect state. Worse, a cache can turn an old success into a fresh-looking authorization decision.

For agents, cache correctness is not only about freshness. It is about whether the result is still trustworthy for this run.

This article builds a small cache contract, shows where to enforce it, and ends with failure-injection tests you can run before putting cached tool results in a long-lived runtime.

The failure mode: fresh enough, wrong authority

Imagine an agent that calls get_invoice and caches the result for five minutes. During those five minutes:

the user loses access to the account;
the tenant policy changes from allow to deny;
the credential is revoked or rotated;
the resource moves to another tenant;
the tool contract changes its meaning;
a previous request completed with an UNKNOWN external outcome.

A normal TTL check answers only “Is this entry old?” It does not answer:

Was this entry produced for the same principal and tenant?
Was it produced under a still-valid policy and credential epoch?
Does the current tool contract interpret the result the same way?
Is the result safe to reuse for this operation, or was it only an observation?
Can the caller prove where the value came from?

Treating those as separate checks prevents a cache hit from silently becoming an authorization bypass.

Define the cache entry as evidence

Do not store only key -> value. Store the context that makes the value valid:

cache_key = hash(
  tool_name,
  normalized_arguments,
  principal_id,
  tenant_id,
  resource_scope,
  policy_version,
  credential_epoch,
  tool_contract_version
)

A useful entry also contains produced_at, expires_at, the producing run ID, a provider request ID, a sensitivity classification, a schema hash, and a revocation or invalidation sequence.

The key prevents accidental cross-context reuse. The metadata lets the read path reject entries that are technically fresh but no longer valid.

Separate observation from authority

The safest default is to cache observations, not permissions.

A cached “invoice total is 42” may be useful for display, but it should not be treated as proof that the current run may refund that invoice. The refund path should perform a current authorization check and, where necessary, a fresh resource read.

A practical classification is:

Class	Example	Reuse rule
OBSERVATION	list of recent logs	Reuse only within scope and freshness bounds
ADVISORY	model-generated summary	Never use as authorization evidence
DECISION_INPUT	account state used for a plan	Revalidate policy and resource version
EFFECT_CONFIRMATION	provider request ID	Reconcile by stable effect key, not TTL

This also protects against using a cache to “remember” that an external write succeeded. A timeout after dispatch is an UNKNOWN outcome. It needs provider reconciliation, not a cache hit that happens to contain a similar response.

Enforce the contract at read time

The read path should reconstruct the current context and reject mismatches before returning a hit.

def usable(entry, current, now):
    if entry is None or entry.expires_at <= now:
        return False

    same_context = (
        entry.principal_id == current.principal_id
        and entry.tenant_id == current.tenant_id
        and entry.resource_scope == current.resource_scope
    )
    same_versions = (
        entry.policy_version == current.policy_version
        and entry.credential_epoch == current.credential_epoch
        and entry.tool_contract_version == current.tool_contract_version
    )

    if not same_context or not same_versions:
        return False
    if current.use == "authorization" and entry.classification != "DECISION_INPUT":
        return False
    if current.use == "effect_confirmation":
        return False  # reconcile by effect key instead
    return True

The invariant is simple:

A cache hit is usable only when its validity context matches the current run and its classification permits the requested use.

Do not rely on the model to decide whether a cached value is safe. The model can request a cache read, but a deterministic layer should decide whether the read is allowed.

Make invalidation explicit

TTL is a fallback. Important changes should invalidate immediately:

policy version increment;
credential epoch increment;
tenant membership or resource ownership change;
tool schema or contract change;
manual incident response;
deletion or retention event;
provider reconciliation that contradicts the cached result.

An append-only invalidation record makes the behavior inspectable:

CREATE TABLE cache_invalidations (
  scope TEXT NOT NULL,
  version INTEGER NOT NULL,
  reason TEXT NOT NULL,
  created_at TIMESTAMP NOT NULL,
  PRIMARY KEY (scope, version)
);

Writers attach the current invalidation version to each entry. Readers reject entries written before the active version for the relevant scope.

That is safer than trying to find and delete every matching key during an incident. Deletion can be incomplete; a version boundary is easy to check and audit.

Prevent stampedes without widening authority

When many runs miss simultaneously, single-flight loading can help, but the lock must not blur security boundaries.

Use a lock key with the same principal, tenant, resource, and version dimensions as the cache key. Never let a privileged request populate a shared entry that an unprivileged request can read.

Bound the wait:

attempt a scoped cache read;
if missing, acquire a scoped single-flight lease;
recheck the cache after acquiring the lease;
load from the provider with current credentials;
validate the response schema and resource scope;
write the entry with its context and invalidation version;
release the lease.

If the provider call times out after dispatch, do not populate a success entry. Record UNKNOWN and reconcile with the provider using a stable request key.

Failure-injection checklist

Add these tests to CI or a staging drill:

change the tenant between write and read;
revoke the credential while the entry is within TTL;
increment policy version while the entry is fresh;
change the tool contract or result schema;
replay identical arguments under a different principal;
return an observation on an authorization path;
crash after provider dispatch but before cache write;
make the provider return an UNKNOWN timeout;
let two tenants contend for normalized arguments;
restore stale cache entries with current policy versions;
fill the cache with an oversized or sensitive result.

For every test, record the expected decision: HIT, MISS, REVALIDATE, DENY, or RECONCILE. A green suite should show not only that the request failed safely, but why.

What to monitor

Track cache behavior as a control-plane signal, not just a latency metric:

hit and miss rate by classification;
rejected hits by reason;
cross-context mismatch attempts;
policy and credential invalidations;
stale-entry reads after restore;
UNKNOWN outcomes awaiting reconciliation;
cache fill failures and provider request IDs;
sensitive-data eviction and retention events.

A rising hit rate can be a bug if rejection and revalidation data disappear. Keep enough evidence to answer which context produced a value and why the current run was allowed to use it.

If you need a managed place to run an always-on OpenClaw or browser-automation workload, managed OpenClaw hosting on Ampere is one option to evaluate. Hosting does not define cache authority, invalidation, credential scope, or reconciliation semantics; those contracts still belong in the application.

Final rule

A TTL answers “how old is this value?” It does not answer “who may trust it now?”

For agent systems, bind cache entries to identity, tenant, resource scope, policy version, credential epoch, and tool contract. Classify what the value is allowed to prove. Invalidate by version when authority changes. Reconcile external effects instead of treating cached responses as confirmation.

That turns caching from a speed optimization into an explicit, testable part of the agent's control plane.

Your AI Agent Upgrade Needs a Compatibility Gate, Not Just a Rollback

Zira — Thu, 20 Aug 2026 03:44:13 +0000

A rollback answers one question: How do I put the old binary back?

It does not answer the harder question: What happens to work, state, credentials, and queued effects that crossed the version boundary while the upgrade was running?

For an AI agent, that boundary is everywhere. A new worker may read an old state row, a queued tool call may use a changed schema, or a restarted process may replay an effect created by the previous release. A clean process rollback can still leave an incompatible state store and duplicate side effects.

This article presents a small compatibility gate you can run before and during an agent-runtime upgrade.

The contract to gate

Give every persisted object and outbound effect an explicit producer contract:

default_version = {
  state_schema: 7,
  tool_contracts: { browser_navigate: 3, repo_apply_patch: 2 },
  effect_protocol: 2,
  credential_epoch: 19
}

The exact format is not important. The important part is that the worker cannot silently assume that the current version is compatible. Persist the versions beside the run, task, queue item, and effect record.

A worker should accept work only when it can prove all of these conditions:

It can read the state schema and preserve fields it does not own.
Its tool-input and tool-output contracts match the queued request.
Its effect protocol understands the existing idempotency and reconciliation states.
Its credential epoch is still authorized for this run.
It can write evidence in a format the recovery tooling understands.

If any check is unknown, pause the item. Do not guess that an old payload is harmless.

Use expand, gate, migrate, contract

A safe rollout is usually four phases:

1. Expand

Deploy readers that understand both the old and new representation. Add new columns or fields without deleting old ones. New writers should continue producing the old representation until the compatibility gate is live.

2. Gate

Put the new worker in a canary mode. It may claim only runs whose contract passes the compatibility test. Keep a visible counter for rejected, unknown, and accepted items.

A useful gate result is more informative than a boolean:

def check_compatibility(item, worker):
    reasons = []
    if item.state_schema not in worker.readable_state_schemas:
        reasons.append("state-schema")
    if item.effect_protocol not in worker.effect_protocols:
        reasons.append("effect-protocol")
    if item.credential_epoch > worker.max_authorized_epoch:
        reasons.append("credential-epoch")
    return {"decision": "reject" if reasons else "accept", "reasons": reasons}

In production code, compare capabilities rather than requiring every version number to be identical. A worker that can safely read schema 6 and 7 should say so explicitly.

3. Migrate

Migrate durable state in resumable, idempotent batches. Record the source version, destination version, migration attempt, and checksum. If a migration stops halfway through, rerunning it should either complete the same transformation or report a deterministic conflict.

Never treat a successful database transaction as proof that an external effect completed. Keep execution state and delivery state separate. An upgrade can commit a dispatch-requested row immediately before the worker dies.

4. Contract

Only after the queue is drained or gated, the state is migrated, and reconciliation is complete should you remove the old reader and writer path. Keep the old path long enough to recover records created before the cutover.

What rollback must preserve

Before declaring a rollback successful, verify these invariants:

Every claimed run has one owner or an explicit expired lease.
Every tool call has a stable idempotency key.
Every outbound effect is CONFIRMED, NOT_SENT, or UNKNOWN, never silently absent.
Every UNKNOWN effect has a provider lookup or a human/operator decision path.
Credential revocation and policy changes apply to both old and new workers.
The evidence writer can still render the run history after the version change.

This is why a rollback should restore a compatible runtime, not merely an older container image.

A failure-injection drill

Run this in a staging environment with a disposable state store:

Queue a task using the old tool contract.
Claim it with the new worker, then kill the worker after the claim transaction.
Change the credential epoch before the worker restarts.
Deploy the old worker and attempt recovery.
Inject a timeout after an outbound provider accepts the request but before your ledger records the response.
Run reconciliation, then inspect whether the task is resumed, paused, or marked unknown.
Repeat with a partially migrated state row and a duplicate delivery attempt.

The expected result is not that everything finishes. The expected result is that every ambiguous transition becomes visible and no stale worker can perform an effect after losing authority.

Hosting is part of the recovery surface

If the agent must stay available during a migration or recovery window, a managed runtime can reduce the operational work of keeping the process online. For teams evaluating that option, managed OpenClaw hosting on Ampere is one hosting path to compare.

That does not provide the compatibility contract for you. You still need durable state, scoped credentials, migration tests, reconciliation, and a rollback plan that accounts for work already in flight.

The checklist

Before upgrading an agent runtime, ask:

Which state schemas can the new worker read and write?
Which queued tool contracts can it execute safely?
What is the effect protocol for requests that are UNKNOWN at the cutover?
How are credential and policy versions rechecked after restart?
Can the migration resume without duplicating a transformation?
Can the old runtime recover records created by the new runtime?
Which evidence proves the cutover was safe?

If you cannot answer those questions, you have a deployment plan and a rollback button, but not a compatibility gate.

The right upgrade test is not whether the new process started. It is whether old and new workers can share the boundary without losing authority, state, or effect evidence.

Your AI Agent's Health Check Lies Unless It Tests the Tool Path

Zira — Wed, 19 Aug 2026 23:42:59 +0000

A process can answer 200 OK while every useful action is failing.

That is the dangerous kind of healthy state for an AI agent. The supervisor sees a live process, the container stays running, and the dashboard is green. Meanwhile the agent cannot reach its model gateway, its tool credentials are expired, its browser profile is locked, or its queue consumer has lost its lease.

The fix is not to make one giant health endpoint that performs a real side effect. The fix is to model health as several explicit signals and test the path that matters without crossing the mutation boundary.

Separate liveness, readiness, and capability

Use at least three states:

Signal	Question	Safe check
Liveness	Is the process event loop responding?	Local response with a deadline
Readiness	Can this worker accept a new run?	Queue lease, config, and dependency checks
Capability	Can this run perform its required tool calls?	Non-mutating probes for the selected capability set

Liveness should be cheap. A liveness failure usually means the process needs a restart.

Readiness is an admission decision. A worker with a stale configuration version or no durable-state connection should stop receiving new work, even if its process is alive.

Capability is run-specific. A text-only task may need a model gateway and a database, while a browser task also needs a valid browser profile, network policy, and download storage. Do not report one global green light when only half the capabilities are usable.

Make the checks typed, not boolean

A boolean loses the information needed for recovery. Return a typed result with an owner and a next action:

{
  "component": "browser",
  "state": "DEGRADED",
  "checked_at": "2026-08-20T01:00:00Z",
  "config_version": "cfg-184",
  "credential_version": "cred-77",
  "evidence": {
    "profile_lock": "held-by-worker-12",
    "read_only_probe": "passed"
  },
  "admission": "reject_browser_runs",
  "recovery": "recycle_profile_after_lease_expiry"
}

Useful states are usually PASS, DEGRADED, FAIL, and UNKNOWN. UNKNOWN matters when a probe timed out or a dependency returned an ambiguous result. It should not silently become PASS, and it should not always trigger an aggressive restart.

A typed result also makes alerts actionable. “Agent unhealthy” is not an operator instruction. “Reject browser runs, keep text runs admitted, recycle the profile after the lease expires” is.

Probe dependencies without performing user mutations

A tool-path probe should be representative but reversible. Examples:

Model gateway: authenticate, request the smallest allowed metadata or token-count operation, then record latency and policy version.
Database: open a connection, verify schema version, and execute a read-only query with a deadline.
Object storage: verify access to the expected bucket and prefix, without uploading a test object unless the system has a cleanup contract.
Browser: launch a disposable profile, navigate only to an allowlisted diagnostic page, and verify the expected browser/CDP handshake.
MCP: list the server’s declared tools and compare them with the policy snapshot, without invoking a mutating tool.

Never use “send an email to test email.” A health check that creates an external side effect is an incident generator. If a provider cannot offer a safe probe, classify that capability as UNVERIFIED and rely on delivery receipts or a separate canary account with an explicit cleanup path.

Tie checks to the run's capability set

Before admission, compute the required capability set from the task, not from the worker's startup profile:

def admit(run, health):
    required = set(run.required_capabilities)
    missing = []

    for capability in required:
        result = health[capability]
        if result.state not in {"PASS", "DEGRADED"}:
            missing.append((capability, result.state))

    if missing:
        return {"decision": "REJECT", "missing": missing}

    return {
        "decision": "ADMIT",
        "health_snapshot": health.snapshot_id,
    }

Store the health snapshot ID with the run admission record. Otherwise an incident review cannot answer a basic question: did the worker admit the task before or after the dependency failed?

The snapshot is not permanent authorization. Recheck policy, credential version, and lease ownership at dispatch time. Health proves a narrow observation at a point in time; it does not grant permission to perform a later action.

Test the failure domains separately

A useful test matrix injects one fault at a time:

Keep the process responsive but block the model gateway. Liveness should pass; model capability should fail; new model runs should be rejected.
Expire the browser credential while the process remains alive. Browser capability should become degraded or failed; text-only work should remain unaffected.
Break the queue lease renewal. Readiness should fail after the lease deadline, and another worker should be able to claim new work.
Return a timeout from a tool dispatch. The result should be UNKNOWN, not an automatic retry that might duplicate a mutation.
Roll the configuration version while a run is waiting. The worker should recheck at dispatch and either refresh safely or reject the run.
Fill the diagnostic storage path. The probe should identify the capacity failure without turning every unrelated capability red.

For each test, record the expected admission decision, the alert, the recovery owner, and whether already-running work is allowed to finish. That last field prevents a readiness failure from becoming an unplanned cancellation storm.

Make recovery narrower than restart

A failed capability does not always justify restarting the whole agent. Prefer the smallest recovery domain:

Refresh a short-lived credential when only its version is invalid.
Recycle a disposable browser profile when the profile lock is stale.
Pause admission when the queue lease is lost.
Route around one unavailable model provider when policy permits.
Restart the process only when liveness or local invariants fail.

For always-on OpenClaw or browser-agent deployments, the hosting layer is part of this recovery contract. A managed runtime such as always-on OpenClaw hosting on Ampere can be a hosting option to evaluate when you need a persistent process and repeatable restart boundary, but it does not remove the need to define state persistence, credential scope, health probes, and recovery policy yourself.

What to put on the dashboard

Show separate panels for:

process liveness and restart count;
readiness and admission rejections;
capability state by dependency and task type;
probe latency and timeout rate;
configuration and credential versions;
unknown outcomes awaiting reconciliation;
recovery actions and their completion status.

The goal is not a prettier green dashboard. The goal is to make “alive but unable to do useful work” impossible to confuse with “ready to accept this run.”

If you build agents, add one tool-path probe this week. Then inject one dependency failure while the process stays alive. If the system cannot reject only the affected work, preserve the evidence, and recover the smallest possible domain, the health check is measuring uptime, not health.

Your AI Agent Needs a Cancellation Contract, Not Just a Stop Button

Zira — Wed, 19 Aug 2026 19:43:13 +0000

A stop button is not a cancellation protocol.

In a toy agent, “stop” can mean setting a boolean and waiting for the loop to exit. In a real agent, work may already be queued, claimed by another worker, inside a browser session, or waiting for an outbound side effect. If cancellation is not represented as durable state, a restart can resurrect work the operator thought they stopped.

The useful question is not “did the process receive SIGTERM?” It is:

Can every layer prove whether this run may still start work, whether in-flight work must finish, and what happened to side effects that were interrupted?

This article turns cancellation into a small contract you can test.

Define cancellation as a state machine

Keep cancellation separate from process liveness. A worker can be alive while its run is cancelled, and a worker can die before it records the cancellation.

A minimal run state machine is:

ACTIVE: new work may be admitted.
CANCELLING: no new work may start; in-flight work is being observed or stopped.
CANCELLED: the run will not resume and no unclaimed step may dispatch.
COMPLETED: the intended work finished.
UNKNOWN: the controller cannot prove whether an external effect happened.

Store the state durably with a monotonically increasing cancel_version:

run_id              status       cancel_version  updated_at
run_42              CANCELLING   3               2026-08-19T12:00:00Z

Workers must carry the version they observed. A dispatch is valid only if the durable row still says ACTIVE with the same version. This closes the race where an operator clicks Stop after a worker checked the run but before it starts a tool call.

Make cancellation explicit at every boundary

A cancellation check only at the top of the agent loop is too weak. Check the contract at each boundary that can create work:

Admission: reject newly submitted steps for a cancelled run.
Queue claim: do not claim a step whose run is CANCELLING or CANCELLED.
Tool dispatch: atomically recheck run state, cancellation version, policy, and credential lease.
Retry scheduling: cancellation revokes future retries, not just the current attempt.
Browser actions: stop before the next navigation or mutation, but record that a page action may already be in flight.
Outbound delivery: apply the delivery policy independently from execution cancellation.

That last distinction matters. Cancelling a code-generation run does not automatically prove that an already-created notification was unsent. Execution and delivery need separate records.

Use cooperative and forced cancellation together

Cooperative cancellation is the default: the worker notices the state change at safe checkpoints and exits cleanly. Forced cancellation is a deadline for the worker that does not cooperate.

A practical sequence is:

ACTIVE
  -> CANCELLING (revoke admission and retries)
  -> drain safe checkpoints
  -> CANCELLED (if no in-flight effects remain)
  -> UNKNOWN (if an external effect cannot be reconciled)

Do not mark a run CANCELLED merely because the worker process exited. A process can die after sending a request and before recording the response. For every external side effect, record an intent with a stable key before dispatch, then reconcile UNKNOWN using the provider’s lookup API, webhook, or an operator decision.

A cancellation timeout should transition the run to UNKNOWN or CANCELLING_TIMEOUT, not silently to success or cancellation. That makes the ambiguity visible instead of converting it into duplicate work on restart.

A small implementation sketch

The critical operation is a compare-and-set, not a read followed by a write:

UPDATE runs
SET status = 'CANCELLING',
    cancel_version = cancel_version + 1,
    updated_at = CURRENT_TIMESTAMP
WHERE run_id = :run_id
  AND status = 'ACTIVE';

A worker dispatch can then require the exact version it observed:

UPDATE steps
SET status = 'DISPATCHED', dispatch_version = :cancel_version
WHERE step_id = :step_id
  AND status = 'CLAIMED'
  AND EXISTS (
    SELECT 1 FROM runs
    WHERE run_id = :run_id
      AND status = 'ACTIVE'
      AND cancel_version = :cancel_version
  );

If the update affects zero rows, the worker must not call the tool. It should release the claim and record CANCELLED_BEFORE_DISPATCH.

Test the races, not just the button

A cancellation feature is incomplete until it survives these injected failures:

Failure	Expected evidence
Cancel between queue claim and dispatch	No tool request, or a reconciled effect record
Worker pauses after the state check	Stale version is rejected at dispatch
Process dies after provider request	Effect becomes UNKNOWN, then reconciles
Retry timer fires after cancellation	Retry is rejected and recorded
Cancellation store is unavailable	Fail closed for new effects; preserve the run as unresolved
Browser action is mid-flight	No next mutation; current action is explicitly unresolved
Controller restarts during drain	Durable CANCELLING state resumes the drain

For each case, assert both safety and evidence: no unauthorized new effect, no lost cancellation intent, and a record an operator can explain later.

Hosting does not define cancellation semantics

If you run an always-on OpenClaw or browser agent, a managed runtime such as managed OpenClaw hosting on Ampere can be one deployment option to evaluate. It does not replace durable run state, fencing, credential scope, or reconciliation. Those remain properties of the agent control plane.

The checklist

Before trusting a Stop button, verify that:

cancellation is durable and versioned;
admission, claim, dispatch, retry, and delivery each recheck it;
workers cannot dispatch with a stale cancellation version;
cooperative drain has a bounded forced-cancellation path;
external effects have stable keys and an UNKNOWN outcome;
restart resumes from durable cancellation state;
tests inject races at every side-effect boundary.

The practical goal is not instant termination. It is a system that can prove what was prevented, what was already in flight, and what still needs reconciliation. That is the difference between a UI button and an operational cancellation contract.

If you are building coding agents or automation that must survive restarts and operator intervention, follow for more concrete control-plane tests and failure drills.

Your AI Agent's Deadline Is Not a Timeout Until Every Worker Shares It

Zira — Wed, 19 Aug 2026 15:43:38 +0000

A timeout is local. A deadline is a contract.

That distinction matters once an AI agent fans work out to queues, workers, browser sessions, MCP servers, or child agents. If every hop invents its own timeout, the system can keep doing work after the user-visible request has already expired. Worse, a retry may outlive the approval, credential lease, or business deadline that made the original call valid.

The practical fix is to propagate one absolute deadline through the execution graph and enforce it at every boundary.

The failure mode

Suppose a request starts at 12:00:00 with a 30-second budget:

The orchestrator waits 30 seconds.
A queue adds 5 seconds of delay.
A worker starts with its own 30-second timeout.
A browser tool retries twice with 10 seconds each.

The user sees a timeout at 12:00:30, but the worker and browser can continue until 12:01:05. That is not just wasted compute. A late tool call can send a message, mutate a record, or use a credential after the request is no longer authorized.

A local timeout answers how long will this process wait? A propagated deadline answers after what instant must every hop stop starting new work?

Use an absolute deadline, not a chain of durations

At ingress, calculate one deadline using a monotonic clock. Pass the deadline, rather than the remaining duration, to every child operation.

from dataclasses import dataclass
import time

@dataclass(frozen=True)
class RunContext:
    run_id: str
    deadline_ns: int
    policy_version: str
    credential_version: str

def remaining_seconds(ctx: RunContext) -> float:
    return max(0.0, (ctx.deadline_ns - time.monotonic_ns()) / 1_000_000_000)

def can_start(ctx: RunContext, reserve_seconds: float = 0.0) -> bool:
    return remaining_seconds(ctx) > reserve_seconds

The wall-clock timestamp is useful for logs, but elapsed-time decisions should use a monotonic clock. Wall clocks can jump because of NTP corrections, VM suspension, daylight-saving changes, or operator adjustments.

A child must not replace the parent deadline with a fresh timeout:

def child_budget(ctx: RunContext, expected_seconds: float) -> float:
    remaining = remaining_seconds(ctx)
    return min(expected_seconds, remaining)

If the remaining budget is zero, the child should be rejected before dispatch. Do not start a call and hope the provider notices the deadline.

Make deadline checks part of dispatch

Check the contract at each side-effect boundary, not only around the top-level request:

Before enqueue: reject work whose deadline is already expired.
Before claim: a worker must not claim expired work.
Before tool dispatch: recheck deadline, policy version, approval, and credential version atomically.
Before retry: calculate the retry reservation, including backoff and provider timeout.
Before outbound delivery: use a separate delivery deadline when the notification is still useful after execution expires.
After an ambiguous result: record UNKNOWN and reconcile with the provider instead of blindly retrying.

The fourth check is commonly missed. A retry with a 2-second provider timeout is unsafe when only 500 milliseconds remain. Backoff is work too, and a retry reservation should include it.

Separate execution and delivery deadlines

A user request can expire while its completion notification is still worth sending. Model those as separate contracts:

{
  "execution_deadline": "monotonic:...",
  "delivery_deadline": "monotonic:...",
  "max_attempts": 2,
  "effect_key": "run-123:completion:v1"
}

Execution expiry means do not begin another tool or model step. Delivery expiry means do not send this notification after it is stale. Keeping the ledgers separate prevents a failed notification from causing the business action to run again.

For a hosted, always-on OpenClaw runtime, managed OpenClaw hosting on Ampere is one deployment option to evaluate. It does not define deadline semantics, cancel already-dispatched side effects, or remove the need for idempotency and credential checks.

Test clock and queue failures deliberately

A useful test matrix should include:

Scenario	Expected result
Queue delay consumes the full budget	Worker refuses to claim
Worker clock is ahead	Monotonic deadline still governs local enforcement
Wall clock jumps backward	No extra execution time is created
Retry reservation exceeds remaining budget	Retry is rejected
Deadline expires during provider call	Outcome is recorded, not guessed
Execution expires but delivery remains valid	Completion is delivered once, if still useful
Worker restarts after expiry	Recovery does not resurrect the run
Duplicate dispatch races with expiry	One effect key and provider reconciliation decide the result

Inject delays at enqueue, claim, pre-dispatch, provider response, and delivery. Record the deadline, remaining budget, clock source, policy version, credential version, and effect key in the evidence trail. Without those fields, a log line saying timed out is not enough to explain whether the work never started, was cancelled, or completed ambiguously.

A compact review checklist

Before shipping an agent workflow, verify:

One ingress deadline is propagated to every child operation.
Duration budgets are derived from the deadline, never reset at a hop.
Monotonic time is used for enforcement and wall time only for display.
Queue claim and side-effect dispatch both recheck expiry.
Retries reserve backoff plus execution time.
Execution and outbound delivery have separate state machines.
Expired or ambiguous effects use stable idempotency keys and reconciliation.
Restarts cannot resurrect work past its deadline.
Tests inject queue delay, clock jumps, provider ambiguity, and duplicate races.

The model can choose the next action, but the runtime must decide whether there is still time and authority to take it. A deadline becomes real only when every queue, worker, tool, and delivery path treats it as the same contract.