DEV Community: Zira

A Coding-Agent Sandbox Is Not a Boundary Until Enforcement Lives Outside It

Zira — Wed, 23 Sep 2026 04:29:30 +0000

A recent Codex sandbox report is a useful reminder for anyone building or operating coding agents: “read-only” is a capability setting, not proof that the host is safe. The security boundary has to be enforced somewhere the agent cannot rewrite, impersonate, or influence from inside the sandbox.

This is not a claim that every sandbox is broken. It is a design test: can the policy decision point be reached through the same process, memory, filesystem, or tool broker that the policy is supposed to constrain?

What was reported

On September 20, BleepingComputer reported two Codex sandbox escapes disclosed by Accomplish AI researchers. The researchers said they reported both issues to OpenAI on August 12 and that both were fixed within eight days. That timeline and the technical details are researcher-reported, with the public incident covered independently by BleepingComputer.

The first issue, called Heapjack by the researchers, affected a JavaScript tool installed by Codex Desktop. Their write-up says the trusted and untrusted V8 contexts shared a Node process and heap. A token used to distinguish trusted code from untrusted code was therefore readable by the untrusted side. The proof of concept forged a request to the native parent process and launched an application outside the Codex process tree.

The striking detail is the operating mode: the researchers say the path worked in read-only, without an approval prompt. BleepingComputer described the attack scenario as opening someone else’s repository in Codex and asking a question about the code. That makes repository provenance part of the threat model, not just the commands the agent is asked to run.

The second issue, called Overpatch, involved the CLI patch tool. The researchers say a patch could name a path such as /tmp to widen the tool’s calculated write permission, then use a symlink to reach a shell startup file outside the workspace. Again, the interesting failure is not “a bad path slipped through.” It is that attacker-controlled patch input influenced the mechanism that computed its own authority.

According to the researchers and BleepingComputer, the fixes were Codex Desktop build 26.818.21641 for Heapjack and Codex CLI 0.149.0 for Overpatch. Treat those as minimums from the report, not as a substitute for checking the current release and your organization’s advisories.

The shared design failure

Both paths have the same shape:

The agent supplied input to a tool.
The tool used that input while deciding what the agent was allowed to do.
The enforcement state was reachable from the same trust domain as the untrusted code.
A successful request crossed into a more privileged process or path.

That is why “the model cannot directly call open” or “the shell cannot write $HOME” is not enough. The real questions are:

Can an untrusted process inspect the authority token, broker pipe, or shared memory used by the trusted side?
Can a path alias, symlink, mount, archive extraction, or parent-directory calculation change the meaning of “inside the workspace”?
Can a tool outside the sandbox be induced to perform the action on the agent’s behalf?
Does a restart preserve a malicious config, hook, plugin, or shell profile?

OpenAI’s current security guidance describes sandboxing and approvals as separate layers: the sandbox limits what commands can technically do, while approval policy controls when Codex asks before acting. The same documentation warns that settings such as full access deliberately widen the trust boundary. That is good operational guidance, but it does not remove the need to test the implementation boundary itself.

A practical test matrix for agent builders

You do not need an exploit reproduction to catch many classes of boundary mistake. Add adversarial tests around the policy decision point.

1. Test repository provenance

Run an untrusted repository in the strictest mode and ask the agent to inspect it. Verify that it cannot:

create or modify files outside the intended root;
launch a host application;
connect to a Docker, SSH, credential, or other privileged Unix socket;
alter global agent configuration; or
cause a trusted helper to perform any of those actions.

Record both the agent-visible result and host-side evidence. A clean chat response is not an audit record.

2. Test path semantics, not only path strings

Exercise symlinks, .. traversal, hard links where applicable, bind mounts, archive extraction, temporary directories, case folding, and platform-specific path forms. Test the parent directory and the final target separately. If a tool calculates permission from a path supplied by the agent, make that calculation happen in a process that cannot be modified by the agent.

3. Separate policy from execution

The component deciding “allowed” should not share mutable authority state with the component executing untrusted code. A stronger pattern is an outer VM or OS boundary, a narrow host broker, and credentials that never enter the guest. The broker should validate an explicit capability request against policy; it should not accept a token or path whose meaning the guest can redefine.

4. Test the recovery boundary

After a failed run, rebuild or reset the environment and check global config, shell startup files, plugin registrations, temporary mounts, and cached credentials. A sandbox that blocks the first action but leaves a persistent hook behind has not recovered cleanly.

5. Treat egress as its own control

Filesystem isolation does not stop data exfiltration if the agent can reach the network. Start with no network, then add narrowly scoped destinations through a policy the guest cannot disable. Keep real credentials outside the guest where possible, and assume anything placed inside an untrusted runtime can be read by code running there.

The rule I would use

The agent may be untrusted. The policy engine must be more trusted than the agent.

If both live in one process, share a heap, share a mutable config, or allow attacker-controlled input to define the permission calculation, call it a defense-in-depth control rather than a hard security boundary.

The useful review question is not “does this feature say sandbox?” It is: what independent mechanism still says no when the agent tries to make the sandbox say yes?

Sources and evidence

Researcher write-up: Escaping the OpenAI Codex sandbox, twice (primary technical disclosure; self-reported)
BleepingComputer coverage (independent technical reporting)
OpenAI Codex security guidance (official product documentation; describes current sandbox, approval, network, and VM guidance)

Visual Studio 2026 BYOK Agent Mode: 6 Tests Before You Trust It

Zira — Wed, 16 Sep 2026 07:10:00 +0000

Microsoft's Visual Studio 2026 September update makes a useful shift in AI-assisted development: the model is becoming a replaceable deployment choice instead of a fixed IDE dependency.

The important part is not simply that Visual Studio can connect to more providers. It is that the new Agent (Preview) changes the runtime contract around those providers.

Microsoft's release notes say the September 18.10 update adds preview BYOK support across Community, Professional, and Enterprise editions. The supported paths include Microsoft Foundry, OpenAI, Anthropic, Ollama, and custom endpoints for OpenAI and Ollama. The same notes also call out a breaking change: BYOK now works with the new Agent (Preview), while the earlier BYOK experience in the previous Ask and Agent modes is no longer supported. Ollama models added in an earlier 18.10 Insider build may need to be added again.

That is a meaningful capability increase, but it is also a reminder that an agent is a control plane, not just a chat box. Before giving a new provider access to a real repository, I would test these six boundaries.

1. Test the capability contract, not just the connection

A provider can connect successfully and still lack important Agent Mode capabilities. Microsoft says unsupported features are marked clearly instead of failing silently. That is better than an invisible fallback, but your workflow should still treat capability discovery as a first-class step.

Create a small smoke repository that exercises the exact actions you expect:

read files across multiple directories
propose and apply a change
run the test suite
inspect a failing test and retry
create a patch or commit
explain which tools were used

Record the result per model and endpoint. A green connection check is not evidence that the provider can perform the full workflow.

2. Separate provider identity from developer identity

BYOK changes where inference happens. It does not automatically answer who is allowed to use the agent, which organization owns the key, or which repository the agent may access.

Use separate credentials for local experiments, team development, and automation. Avoid putting a long-lived provider key in a repository, shell history, or a shared machine image. If the provider supports project-level keys or spend limits, use them. The useful audit record is not just “the agent ran”; it is “which principal, endpoint, model, repository, and tool policy authorized this run?”

3. Make tool permissions explicit

Agent Mode can be valuable precisely because it can do more than generate text. That also makes broad permissions dangerous. Start with a repository-specific working directory and a deny-by-default policy for unrelated paths, network access, package installation, and credential files.

A simple policy sketch might look like this:

agent:
  repository: ./sample-repo
  allow:
    - read:src/**
    - write:src/**
    - run:dotnet test
  deny:
    - read:.env*
    - read:**/*credential*
    - write:.github/workflows/**
    - network:*
  approval_required:
    - package_install
    - git_push
    - database_migration

The exact format will vary by harness. The point is to make the boundary reviewable before the model sees the repository.

4. Preserve evidence outside the agent's control

If the agent edits the same files that contain its own explanation, those files are not a complete audit trail. Capture the request, selected endpoint, model identifier, tool calls, diffs, test output, approvals, and final status in a location the agent cannot rewrite retroactively.

Do not log secrets or full prompts that contain credentials. A useful minimum is a run ID, repository commit, policy version, provider, model, timestamps, tool results, and hashes of the resulting patch.

5. Test side effects and interruption

Run the agent against a disposable branch and interrupt it at several points: during a file edit, during a test run, and after a commit but before the final response.

Then answer two different questions:

Can the execution resume without losing state?
Can the workflow prove whether a side effect already happened?

Those are not the same failure. A restart may recover the conversation while still repeating a commit, package install, migration, or external request. Use checkpoints and idempotency keys where the workflow can create durable side effects.

6. Treat preview upgrades as contract changes

The release notes explicitly describe the new BYOK path as preview and document breaking changes between Insider builds. That is enough reason to pin the IDE and provider configuration for repeatable work, keep a known-good fallback, and retest after upgrades.

For Ollama in particular, Microsoft says models added in an earlier 18.10 Insider build may need to be added again in the next Insider build. That is a small migration detail, but it is exactly the sort of detail that turns a “model outage” into a confusing afternoon if it is not in the upgrade checklist.

The practical takeaway

Visual Studio's BYOK Agent Preview is interesting because it makes model choice more flexible. The engineering challenge is making that flexibility observable and reversible.

Before adopting it for a production repository, keep a small compatibility matrix:

Provider/model	Agent capabilities	Tool policy	Test result	Rollback path
endpoint A	pass/fail per action	policy v3	commit + tests	endpoint B
endpoint B	pass/fail per action	policy v3	interrupted run	pinned IDE build

This is a vendor-documented preview, not independent proof of reliability or productivity. The right first experiment is therefore not a benchmark screenshot. It is a bounded repository, explicit permissions, an external audit trail, and a recovery test you can repeat after the next update.

Source: Visual Studio 2026 release notes, including the September 8, 2026 18.10 update and the September 15, 2026 18.10.1 patch.

Codex CLI 0.153: Remote Plugins Make Supply-Chain Policy Part of Your Agent Runtime

Zira — Thu, 03 Sep 2026 04:25:25 +0000

Codex CLI 0.153.0 is not just a terminal UI update. It adds a plugin command that can list, install, and remove plugins from remote marketplaces, while also improving how sessions recover after an app-server disconnect.

That combination changes the operational question from “Can the agent call a tool?” to “Which extension code is allowed to enter this runtime, and what state survives when the runtime reconnects?”

What changed in Codex CLI 0.153

The official 0.153.0 release adds:

A plugin CLI for listing, installing, and removing plugins from remote marketplaces.
Automatic TUI reconnection after an external app-server connection drop, preserving drafts and transcripts while pausing uncertain or queued submissions for review.
Guardian review history that survives compaction, restarts, and user-created forks, within rollback boundaries and isolated subagent history.
More complete TUI history, including patches, background-terminal input, and individual completed commands.
A tui.auto_recap = false setting for teams that prefer manual /recap control.

The release notes are the authoritative source for these features. AI/TLDR’s release summary is useful for discovery, but it is a secondary summary, not independent performance evidence.

Remote plugins are a new trust boundary

A marketplace client reduces the friction of extending an agent. It also makes installation provenance part of the runtime’s security model.

Before allowing a plugin into a development environment, record at least:

The marketplace and exact source URL.
The plugin version and immutable commit or digest, if available.
The tools, filesystem paths, network destinations, and credentials it can reach.
The review status and the person or policy that approved it.
A removal and rollback path that does not depend on the plugin itself.

A name-based allowlist is not enough. Two versions of the same plugin can have different behavior, and a plugin that only appears to “format code” may still be able to read files or invoke a shell through its host integration.

A minimal installation record should preserve the plugin name, marketplace URL, exact version, immutable digest, effective capabilities, approver, and expiry. Store that record outside the plugin so removal and rollback do not depend on the plugin itself.

The digest and capability fields are the important parts of this policy shape. They are recommendations, not claims about the Codex marketplace schema.

Reconnect is not the same as resume

The new reconnect behavior is valuable because a dropped app-server connection no longer necessarily destroys the draft or transcript. But preserving state does not prove that an interrupted tool submission did not reach its destination.

For every side-effecting tool call, keep execution state separate from delivery state:

Prepared: arguments validated, no external call started.
In flight: the request may have reached the destination.
Accepted: the destination returned a durable identifier or equivalent evidence.
Unknown: the client lost the response after sending.
Reconciled: a separate read confirmed the external state.

After reconnect, an “unknown” call should not be blindly replayed. Use an idempotency key where the destination supports one, or reconcile by querying an external identifier before retrying. This is the difference between recovering the agent session and recovering the real-world side effect.

A practical upgrade checklist

If you are upgrading a long-running coding-agent environment:

Pin the CLI version and capture the release commit.
Test plugin installation in a disposable workspace first.
Inspect the plugin’s effective filesystem, network, and credential reach.
Require review for new plugin sources and record approval with an expiry.
Kill the app-server during a draft and during a tool call.
Confirm that drafts and transcripts return after reconnect.
Confirm that queued or uncertain submissions remain paused.
Reconcile external state before retrying any side effect.
Verify that rollback removes the plugin and its persisted state.

The release improves the recovery path, but it does not eliminate the need for capability boundaries, idempotency, or recovery testing. Those remain properties of the surrounding deployment and the tool destinations.

Bottom line

Codex CLI 0.153 makes two parts of agent infrastructure more explicit: extension provenance and session recovery. Treat remote plugins as code entering a privileged runtime, and treat reconnect as state recovery rather than proof that every external action is safe to repeat.

For teams that need an always-on place to run an agent, Ampere can be a contextual hosting option, but hosting does not replace plugin review, credential scoping, idempotency, or restore testing.

Primary source: OpenAI Codex CLI 0.153.0 release notes.

Your AI Coding Agent Can Edit Compose UI. Can It Verify the Result?

Zira — Wed, 02 Sep 2026 22:18:49 +0000

AI coding agents are good at producing UI code and surprisingly bad at proving that the UI actually works. A compile pass is not the same as a rendered screen, a correct semantic tree, or a click that reaches the intended handler.

Compose Multiplatform 1.12.0 adds an experimental MCP server to Compose Hot Reload. It gives an AI coding agent a feedback path into a running Compose application: trigger a reload, take a screenshot, inspect the semantic tree, simulate clicks and text input, and read application logs.

That changes the useful question from “Can the agent write this composable?” to “Can the agent observe the result of its edit and detect a broken interaction?”

What the Compose Hot Reload MCP server exposes

According to JetBrains’ release announcement, the experimental server connects an agent to a running app and supports:

triggering reloads after an edit
taking screenshots of the rendered UI
inspecting the semantic tree
simulating clicks and text input
reading application logs

The Compose Hot Reload documentation contains the connection details and the current tool list. The Compose Multiplatform 1.12.0 release notes are the versioned reference for the feature.

This is an observation loop, not an automatic correctness proof. A screenshot can show that a button is visible, while the semantic tree and logs can provide stronger evidence about labels, roles, and runtime failures. None of those alone proves that the feature satisfies the product requirement.

A practical agent verification loop

A useful harness should make the agent follow an explicit sequence rather than letting it stop after compilation:

Edit one bounded change. Keep the diff small enough that a failed observation has a plausible cause.
Reload the running app. Record whether reload completed or timed out.
Check the semantic tree. Assert that required nodes exist with the expected labels or roles.
Capture a screenshot. Use it to catch layout regressions, missing content, or an unexpected screen state.
Exercise one critical interaction. Simulate the click or text entry that matters for the changed path.
Read logs after the interaction. Treat new exceptions or error-level entries as a failed run.
Store evidence with the change. Keep the reload result, assertions, screenshot reference, and relevant logs attached to the agent run.

The important design choice is to keep these as separate checks. “The screenshot looks right” should not silently substitute for “the semantic node exists,” and “the reload succeeded” should not substitute for “the interaction produced the expected state transition.”

Where this helps most

The MCP server is especially useful for UI changes with a short, deterministic path:

a form field must accept input and update state
a button must become enabled after validation
a navigation action must render a specific destination
a loading or error state must appear under a controlled condition
a regression must be reproduced from a known starting state

For each path, define an observable contract. For example: after entering a valid email and clicking Continue, the semantic tree contains the next-screen heading, the log contains no new exception, and the screenshot shows the expected state.

The tradeoffs and the safety boundary

This is an experimental developer tool, so it should be treated as a test harness rather than a production control surface. An agent that can simulate input and inspect a running app has meaningful capabilities. Run it against a disposable or tightly scoped environment, avoid real credentials and production data, and keep the available actions limited to the application under test.

There is also a determinism problem. Screenshots and logs can vary with timing, platform, fonts, network state, and background work. Prefer assertions over stable semantic properties and explicit state transitions. Use screenshots as supporting evidence, not as the only oracle.

Finally, separate execution evidence from product correctness. The tool can show what happened in the running app; it cannot decide whether the feature is desirable, safe to ship, or compliant with a business requirement. Those remain separate review gates.

Bottom line

Compose Hot Reload’s experimental MCP server is valuable because it closes part of the loop between an AI coding agent and the UI it changes. The practical benefit is not that the agent becomes trustworthy by default. It is that the harness can require observable evidence before accepting a UI edit.

The strongest workflow is therefore: small diff, reload, semantic assertions, targeted interaction, log check, and persisted evidence. Give the agent a way to see the result, then make the acceptance contract stricter than “it compiled.”

AI Coding Agent Security: Why a Git Folder Can Run Code Before Approval

Zira — Wed, 02 Sep 2026 11:00:09 +0000

A new class of AI coding agent security bug has a surprisingly ordinary entry point: Git itself.

Manifold Security calls the issue GitSpawn. The core problem is not that an agent misunderstood a prompt. It is that an agent ran a normal Git command while gathering repository context, and Git honored an executable setting stored inside the repository's own .git/config.

That turns “open this project” into a potentially privileged operation.

The mechanism: context gathering becomes code execution

Many CLI coding agents inspect a repository as soon as they start. They may run commands such as:

git status --porcelain=2 --branch
git diff --name-only HEAD

These are ordinary commands, but Git may refresh its index before returning the result. Git's core.fsmonitor setting can name a helper program to run during that refresh. The setting is read from the repository-local .git/config file.

A project delivered as a complete directory, ZIP archive, shared-drive folder, USB copy, or sync-folder snapshot can therefore carry a Git configuration that names a program. When the agent runs Git, the helper can execute with the developer's operating-system privileges.

The important boundary is timing. Manifold reports that, in the affected flows it tested, execution happened during background context gathering, before a workspace-trust prompt, before an approval prompt, and in some cases before authentication. The model does not need to generate a dangerous command, and the user may not see a prompt.

What the disclosure actually found

This is a researcher disclosure, not an independent prevalence study. Manifold reports eight findings across seven agents and says four remained unpatched at publication. Its September 1 update says OpenAI Codex and Cursor were also affected and had since been patched.

The report lists these statuses at publication:

Claude Code's core.fsmonitor path: fixed in 2.1.196.
Goose: fixed in 1.44.0, with CVE-2026-72718.
OpenAI Codex and Cursor: patched, according to the update.
Hermes Agent: still unpatched in the tested 0.21.0 release, with CVE-2026-71963.
Qwen Code: still unpatched in the tested 0.22.3 release.
Grok Build: still unpatched in the tested 1.0.13 release.
Claude Code's separate ultrareview path: reported as still unpatched in the tested 2.1.252 release.

Treat those version claims as time-sensitive. Check the vendor's current release notes and advisory before deciding whether a version is safe.

Why a sandbox may not save you

A sandbox around the model's tool calls is not automatically a sandbox around every subprocess the agent launches. In this case, the dangerous process is Git, which the agent invokes for its own bookkeeping. If the product does not sanitize repository configuration for that subprocess, the command can run outside the model-facing approval path.

That is a useful design lesson for agent harnesses: permission checks must cover the complete execution graph, including startup probes, Git helpers, formatters, language servers, hooks, plugins, MCP servers, and other “read-only” context tools.

A safer intake workflow

Before opening an unfamiliar directory with an AI coding agent:

Prefer cloning from a trusted remote over accepting a complete project folder. A normal clone does not carry the remote repository's original .git/config.
Inspect the local Git configuration without launching the agent:

sed -n '1,200p' .git/config
Treat any Git setting that names an executable as untrusted code. The reported proof of concept uses core.fsmonitor, but the broader rule is more useful than memorizing one key.
Use a disposable VM or tightly scoped container for untrusted project directories. Keep SSH keys, cloud credentials, and production tokens out of that environment.
Update the agent and verify the exact fixed version. “The agent has an approval prompt” is not enough if startup context gathering happens before the prompt.

For agent authors, the mitigation is equally concrete: pass safe Git configuration explicitly for background calls, for example git -c core.fsmonitor=false status, and review every subprocess used before trust and authorization decisions.

The broader takeaway

AI coding agent security is partly a model problem, but it is also a plumbing problem. The model can be perfectly aligned while the harness exposes a trusted developer process to untrusted repository metadata.

The right review question is not only “What commands can the agent propose?” It is also “What runs automatically before the user has approved anything?” Audit that startup path, test it with hostile-but-non-destructive fixtures, and make execution, approval, and sandbox boundaries line up.

Sources and evidence provenance

Manifold Security: Coding Agents Hijacked by a Git Call — primary researcher disclosure; version and patch status are the researcher's reported findings.
AI/TLDR summary of GitSpawn — secondary tracker summary, cross-checked against the primary report.
Git documentation for core.fsmonitor — upstream reference for the Git configuration behavior.

This article intentionally avoids reproducing a weaponized repository or payload. The goal is to make the boundary visible and give developers a safer intake and testing workflow.

Codex CLI 0.152: The Small Release That Changes Agent Control

Zira — Wed, 02 Sep 2026 05:00:09 +0000

OpenAI’s Codex CLI 0.152.0 release is easy to read as a routine feature update. The more useful reading is operational: it changes how a coding agent plans, consumes MCP output, handles long-running commands, and protects credentials during cloud-task requests.\n\nThis is not a benchmark claim. It is a release-level change list, and the practical impact depends on how you run Codex.\n\n## 1. Planning is now opt-in\n\nThe release disables the update_plan tool by default. If your workflow depends on explicit plans, enable it with:\n\n tools.update_plan.enabled = true\n\nThat is a small configuration change with a large workflow implication. A plan is not the same thing as execution state. If you use plans for review, handoff, or recovery after interruption, store the important state somewhere your process can recover rather than relying on a transient conversation view.\n\n## 2. MCP output gets a per-tool budget\n\nCodex 0.152.0 adds an output_token_limit setting for individual MCP tools, and the release notes say truncation is consistent across session resumes. This is useful because one verbose tool can otherwise crowd out the context needed for the actual coding task.\n\nA practical starting point is to set limits by tool role:\n\n- Small limits for status and metadata tools\n- Moderate limits for search and diagnostics\n- Larger, explicitly reviewed limits for logs or generated reports\n\nThe tradeoff is obvious: truncation controls context pressure, but it can also hide the line that explains a failure. Tool output should therefore carry a cursor, artifact link, or continuation mechanism instead of silently dropping information.\n\n## 3. Long commands need an explicit deadline\n\nApp-server clients can configure thread/shellCommand timeouts, including deadlines longer than one hour. That helps with builds, test suites, and migrations, but a longer timeout is not a cancellation contract. A production runner still needs progress reporting, cancellation behavior, retry rules, and a durable record of whether the command actually completed.\n\n## 4. Credential protection is part of the agent feature set\n\nThe release notes also say cloud task requests reject untrusted backend URLs and disable redirects. That matters because redirects are not just a networking detail when a task can access saved credentials. A safer request path needs an explicit origin policy, and the policy should be tested with redirects, DNS changes, expired credentials, and resumed sessions.\n\n## Where a hosted runtime fits\n\nIf Codex is part of an always-on remote workflow, a hosted runtime such as Ampere can be a convenient place to keep the process and its surrounding automation available. That does not remove the need for state contracts, scoped credentials, isolation, idempotency, or recovery tests. Hosting solves a runtime-placement problem; it does not automatically solve the control-plane problem.\n\n## A practical upgrade checklist\n\nAfter upgrading to 0.152.0, I would test four things before trusting a long-running workflow:\n\n1. Start a task with planning enabled and confirm the setting survives a new session.\n2. Send oversized output through each important MCP tool and verify truncation is visible and recoverable.\n3. Interrupt and resume a long shell command, checking whether execution state is unambiguous.\n4. Attempt a cloud-task redirect to an untrusted origin and confirm it is rejected without exposing credentials.\n\nCodex CLI 0.152.0 is therefore less about a flashy new capability and more about making agent behavior bounded and inspectable. Those defaults are worth reviewing before treating an upgrade as a drop-in replacement.\n\n### Sources\n\n- OpenAI Codex CLI 0.152.0 release notes\n- OpenAI Codex CLI changelog\n- AI/TLDR release summary

Your Agent Gateway Needs Failure-Domain Tests, Not Just a Health Check

Zira — Mon, 24 Aug 2026 01:41:56 +0000

A green health check can coexist with a broken agent.

If you are trying to keep an OpenClaw agent available without owning the server setup, Ampere is a free hosting option to evaluate: https://ampere.sh. It can reduce the infrastructure work, but it does not replace durable state, scoped credentials, isolation, or recovery testing.

The process may be listening, the dashboard may say "healthy", and yet:

a second inbound message disappears before it enters a queue;
a gateway restart loses a suspended tool call;
a background run reaches "completed" but its announcement never arrives.

Those are not one availability problem. They are three different recovery contracts.

A useful field report on OpenClaw issue #128067 describes all three symptoms in one deployment. This is practitioner-reported evidence, not an independent benchmark, but it suggests a practical test plan: stop asking whether the gateway is alive and test whether each failure domain preserves its contract.

Model the states separately

Do not collapse everything into a single done boolean. Keep at least these states:

accepted: the ingress boundary assigned a durable request ID;
queued: a queue record exists and can be recovered;
running: a worker owns the execution lease;
completed: execution produced a recorded outcome;
delivered: the outbound provider acknowledged the message;
unknown: the process died during a side-effect boundary and reconciliation is required;
failed: the system has a terminal, evidenced failure.

A run can be completed while delivery is unknown. Treating those as the same state is how operators report success when the user saw nothing.

Use stable IDs for both the request and each outbound effect:

request_id = uuid()
effect_id = hash(request_id + "completion-announcement")

The exact storage technology is less important than the invariant: a restart must not create a second logical request or a second outbound effect just because the first process disappeared.

Test 1: ingress durability

Send two messages with a small gap, then inspect the durable boundary rather than the chat UI.

send(message_a)
send(message_b)

assert distinct(request_id_a, request_id_b)
assert queue.contains(request_id_a)
assert queue.contains(request_id_b)
assert queue.order_is_observable(request_id_a, request_id_b)

Inject failures between receipt and queue commit:

kill the gateway after parsing but before the insert;
delay the database response after the insert;
retry the same delivery with the same provider event ID;
deliver two messages concurrently.

For every case, the result should be one of two explicit outcomes: the request is durably accepted exactly once, or the caller receives a retryable failure. "No queue entry and no error" is not an acceptable state.

Test 2: execution recovery

Start a tool call that takes longer than the gateway process. Terminate the process while the call is pending, then restart it.

The recovery contract must say which of these is true:

the tool call can be safely resumed;
the call is cancelled and the agent compensates;
the outcome is unknown and a reconciler checks the provider;
the operation is deliberately non-replayable and needs human review.

Do not infer completion from a worker log line. Persist a phase transition before dispatch and persist the provider result after the call. A crash between those writes is exactly the boundary your test must exercise.

For non-idempotent tools, a stable effect key and provider-side lookup are safer than blind replay. If the provider has no lookup API, quarantine the operation as unknown instead of guessing.

Test 3: outbound delivery recovery

Execution and delivery need separate ledgers. Simulate a crash after the announcement is accepted by the provider but before your worker records the acknowledgement.

if delivery.status == "unknown":
    provider_result = lookup_by_idempotency_key(effect_id)
    if provider_result.sent:
        mark_delivered(effect_id, provider_result.message_id)
    else:
        retry_once_with_same_effect_id(effect_id)

Then test the inverse boundary: the worker records success, but the network response is lost. The next process must reconcile before sending a duplicate.

This is especially important for spawned jobs, cron tasks, and announce-style messages. A queue dashboard showing pending tells you little unless you also know whether the consumer attempted delivery, whether the provider accepted it, and whether the result was reconciled after a restart.

Test 4: background-work truthfulness

Run a scheduled job that completes its internal work but cannot publish its result. Verify that the UI and alerting surface say:

execution: completed;
delivery: failed or unknown;
recovery action: retry, reconcile, or human review.

A single green check should never hide a red delivery state.

Track these counters separately:

accepted requests;
queue insert failures;
lease expirations;
executions recovered after restart;
unknown effects awaiting reconciliation;
deliveries acknowledged by the provider;
duplicate attempts suppressed by the effect key.

A compact failure matrix

Boundary	Injected failure	Required evidence
ingress to queue	process kill	durable request ID or retryable error
queue to worker	lease expiry	one owner or explicit requeue
worker to tool	process kill	resume, compensate, or `unknown`
tool to ledger	lost response	provider lookup or quarantine
ledger to outbound provider	network loss	same effect key, no blind duplicate
provider ack to UI	callback loss	delivery reconciliation and alert

Run this matrix on a clean environment and after every change to queueing, worker leases, provider adapters, or gateway restart behavior. If you run an always-on OpenClaw deployment, hosting is only one layer of the system and does not supply these state contracts. You still need the ledgers, idempotency keys, isolation, and recovery tests.

The practical definition of healthy

A healthy agent gateway is not merely a live process. It can prove, for every request:

where the request became durable;
which worker owned execution;
what happened at each tool boundary;
whether outbound delivery was acknowledged;
what recovery action is safe after a crash.

Start with one failure injection per boundary. Make the evidence queryable. Then make the health check fail when a contract is violated, not only when the process stops responding.

That is the difference between monitoring uptime and testing whether the agent can be trusted unattended.

Two AI Agents Read One Row. How to Stop the Lost Update

Zira — Sat, 22 Aug 2026 21:36:55 +0000

A version check is not automatically optimistic concurrency control.

If an agent reads a row, reasons for a few seconds, and then writes a decision, another worker can change that row in the read-to-write gap. The final UPDATE may still return success while silently overwriting newer state.

This is especially easy to miss in agent workflows because the reasoning step is slow, retries are common, and the write often looks like an ordinary CRUD operation.

This post builds a small failure lab and turns the race into a testable contract.

The bug: the check happened too early

A tempting implementation is:

grep -n "version" worker.py

Read the row and its version.
Check that the version is acceptable.
Ask the model what to do.
Update the row later.

The check in step 2 says nothing about whether the row is still current in step 4. Two workers can both read version 7, both produce valid decisions, and both write. The second write wins even though its input was stale.

The failure is not necessarily a database error. It is a successful write based on an invalid snapshot.

Make freshness part of the write predicate

Use a monotonic version, and require the version observed by the worker to match the version being replaced:

action = db.execute(
    """
    UPDATE agent_tasks
       SET status = :new_status,
           decision = :decision,
           version = version + 1,
           updated_at = CURRENT_TIMESTAMP
     WHERE task_id = :task_id
       AND version = :expected_version
       AND status = 'ready'
    """,
    {
        "task_id": task_id,
        "expected_version": observed_version,
        "new_status": new_status,
        "decision": decision,
    },
)

if action.rowcount != 1:
    raise StalePlan("the row changed after the agent read it")

The important property is that the read and the model call do not need to be inside one long database transaction. The final compare-and-set write is the fence.

A zero-row update is not a generic retry signal. It means the plan was produced from a snapshot that no longer owns the right to change the row.

Keep the plan tied to the snapshot

Do not pass only a task ID into the write path. Persist the inputs that made the decision:

CREATE TABLE agent_plans (
  plan_id TEXT PRIMARY KEY,
  task_id TEXT NOT NULL,
  observed_version INTEGER NOT NULL,
  input_hash TEXT NOT NULL,
  decision_json TEXT NOT NULL,
  state TEXT NOT NULL CHECK (state IN ('proposed', 'applied', 'stale', 'unknown')),
  created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
);

The input hash should cover the normalized state the model actually saw, not just the database primary key. If the prompt included policy, tool availability, or a selected artifact, include their versions too.

At apply time, verify both the row version and the plan state. A plan that was already applied must not be applied a second time, even if a client retries after a timeout.

A minimal failure-injection test

You can reproduce the race without an LLM. Replace the reasoning step with a barrier so both workers pause after reading the same version:

async def worker(barrier, task_id):
    row = await read_task(task_id)
    await barrier.wait()  # both workers now hold the same snapshot
    decision = {"owner": current_worker_id()}
    return await apply_plan(
        task_id=task_id,
        observed_version=row.version,
        decision=decision,
    )

The expected result is exactly one successful apply and one stale-plan result. If both calls report success, the write fence is missing or the test is not exercising the real update path.

Run the lab with these cases:

Case	Expected result
Two workers read version 7	One applies; one is stale
Worker retries the same plan	No second business effect
Row changes while the model is thinking	Apply is rejected
Policy version changes	Plan is rejected or re-planned
Database response is lost after commit	Outcome becomes UNKNOWN and is reconciled
Worker restarts after apply	Recovery does not apply the plan again

Treat timeout after write as UNKNOWN

A client timeout does not prove that the UPDATE failed. The database may have committed before the connection broke.

Give every apply attempt a stable effect key, such as task_id plus plan_id, and record it in a durable effect ledger. On a timeout:

Look up the effect key.
Compare the stored plan and row version.
Mark the outcome applied, stale, or UNKNOWN.
Reconcile UNKNOWN before allowing another attempt.

Never solve an ambiguous write by blindly rerunning the model and applying its new answer. That can turn one uncertain side effect into two different decisions.

What to monitor

A useful dashboard separates these signals:

stale-plan rejects, by tool and workflow;
plans that waited longer than their snapshot budget;
duplicate apply attempts by effect key;
UNKNOWN outcomes and reconciliation age;
policy or tool-contract changes between read and apply;
successful writes that had no matching plan record.

The last metric should be zero. If it is not, some path can mutate agent state without the same evidence and fencing rules.

Checklist

Before trusting an agent that writes shared state, verify:

The model decision carries the exact version and policy inputs it observed.
The final write uses an atomic compare-and-set predicate.
A zero-row update becomes a typed stale-plan result, not a generic retry.
Every business effect has a stable idempotency key.
Lost responses are reconciled instead of replayed blindly.
A barrier test proves two concurrent readers cannot both apply version 7.
Restarts and partial failures are included in the test matrix.

A version column is useful, but the real boundary is the version check at the write. Without that fence, an agent can be perfectly correct about an old snapshot and still corrupt the current one.

Your AI Agent Returned HTTP 200. Why Did the Workflow Still Fail?

Zira — Fri, 21 Aug 2026 13:34:21 +0000

A successful HTTP response is not a successful agent run.

A recent practitioner report from a 58-day deployment of 78 agents recorded 6,768 failed outputs. The failures were not transport errors: every one returned HTTP 200, had plausible length, and looked fluent. The most expensive failures were boring shape mismatches: missing required fields, wrong language, forbidden phrases, or an answer for a different stage.

That is a useful warning for anyone building coding agents, review agents, or unattended automation:

Treat the model response as untrusted data. Validate the contract at the boundary before another stage can consume it.

This post turns that observation into a small, reproducible failure lab.

The failure you should be able to reproduce

Imagine a review stage whose downstream parser expects a verdict line:

action: approve

A model can return a thoughtful review with the verdict buried in prose. A human approves it. A parser does not.

The transport layer is green. The model call is green. The workflow is broken.

The same class of failure appears when:

a JSON field is present but has the wrong type
a response is in the wrong language
a tool returns an error string inside a successful content envelope
a stage emits output, but the next stage never reads that artifact
a reviewer from the same model family approves a shared blind spot

These are not reasons to add a larger model first. They are reasons to make the boundary observable and enforceable.

Build a contract gate

Start with deterministic checks that do not ask an LLM to judge another LLM.

def validate_review(text: str) -> list[str]:
    errors = []

    if len(text.strip()) < 150:
        errors.append('too_short')

    if not any(line.startswith('判定:') or line.startswith('判定：')
               for line in text.splitlines()):
        errors.append('missing_required_verdict')

    forbidden = ['お客様の声', '顧客の声']
    if any(term in text for term in forbidden):
        errors.append('forbidden_phrase')

    if not any('。' in line for line in text.splitlines()):
        errors.append('expected_language_missing')

    return errors

errors = validate_review(model_output)
if errors:
    record_rejected_output(errors, model_output)
    stop_downstream_dispatch()
else:
    publish_to_next_stage(model_output)

The important part is not the exact Japanese check. Replace it with the contract your system actually needs: required headings, schema types, repository paths, test names, citation fields, or a bounded action list.

A gate should return structured evidence, not only true or false:

action: reject
reasons:
  - missing_required_verdict
  - forbidden_phrase
contract_version: review-v3
artifact_id: art_01J...

That makes a failure repairable instead of turning it into a green dashboard with a missing deliverable.

Record why, not just that

A common anti-pattern is storing only a boolean such as contract_satisfied = false. That destroys the information needed to debug drift.

Store at least:

Field	Why it matters
artifact_id	Connects the output to its producer and consumer
contract_version	Shows which rules were active
observed_checks	Proves what was actually tested
failure_reasons	Separates shape, language, policy, and transport failures
raw_output_hash	Allows correlation without exposing sensitive content
downstream_read_at	Detects outputs that nobody consumed
reviewer_family	Exposes correlated writer/reviewer blind spots

Do not silently discard rejected output. Apply retention and redaction rules, but preserve enough evidence to answer: what was produced, which contract rejected it, and did any later stage read it?

This is the same evidence discipline I use in audit-ready agent logs: an event saying “run completed” is weaker than a record of the checks and artifacts that made completion meaningful.

Add an artifact-lineage check

One surprising failure mode is a healthy upstream stage whose output is never used. Test this explicitly.

Give every produced artifact a stable ID.
Require the consumer to record the input artifact ID.
Reject a stage that claims success without a consumed input ID.
Compare produced and consumed counts over a time window.
Alert on a growing gap, even if every process heartbeat is green.

This catches wiring bugs that output-quality checks cannot see.

Run the failure lab

Before trusting a new agent workflow, inject each case and verify the expected evidence:

Injection	Expected result
Remove the required verdict line	Reject before downstream dispatch
Return valid-length text in the wrong language	Reject with language evidence
Put an error string in a successful tool envelope	Mark the tool call failed
Drop the artifact ID between stages	Block consumption and alert on lineage gap
Change the contract version mid-run	Revalidate or move the run to UNKNOWN
Make writer and reviewer share a known blind spot	Require an independent check or human review
Crash after provider acceptance but before ledger write	Reconcile before retrying

The last case matters for side effects. A contract gate protects output shape; it does not prove that an external action did or did not happen. Keep execution evidence and outbound-delivery evidence separate.

Where hosting fits

An always-on runtime can keep schedulers, workers, and evidence writers available, but hosting does not define your output contract or make a green HTTP response meaningful. If you need managed infrastructure for an unattended OpenClaw workload, managed OpenClaw hosting on Ampere is one option to evaluate. You still own validation, credential scope, prompt-injection defenses, and reconciliation.

The practical checklist

Before shipping an agent stage, verify that:

transport success is distinct from contract success
required fields and language are machine-checked
rejection reasons are durable and queryable
artifact lineage connects producers to consumers
the reviewer cannot share every blind spot with the writer
output quality and side-effect completion are separate states
failure injection covers both malformed output and crash windows
dashboards count produced and consumed artifacts, not only invocations

The question is not “did the model answer?” It is “did a versioned, observable contract accept an artifact that the next stage actually consumed?”

That is the difference between an agent that is alive and a workflow that is working.

If you build AI agents or developer tooling, follow me for practical failure labs and reproducible control-boundary tests rather than capability demos.

Your AI Agent Cache Needs a Trust Boundary, Not Just a TTL

Zira — Thu, 20 Aug 2026 07:46:45 +0000

A short TTL does not make an agent's cached tool result safe.

A result can still be wrong for the current user, tenant, credential epoch, policy version, or side-effect state. Worse, a cache can turn an old success into a fresh-looking authorization decision.

For agents, cache correctness is not only about freshness. It is about whether the result is still trustworthy for this run.

This article builds a small cache contract, shows where to enforce it, and ends with failure-injection tests you can run before putting cached tool results in a long-lived runtime.

The failure mode: fresh enough, wrong authority

Imagine an agent that calls get_invoice and caches the result for five minutes. During those five minutes:

the user loses access to the account;
the tenant policy changes from allow to deny;
the credential is revoked or rotated;
the resource moves to another tenant;
the tool contract changes its meaning;
a previous request completed with an UNKNOWN external outcome.

A normal TTL check answers only “Is this entry old?” It does not answer:

Was this entry produced for the same principal and tenant?
Was it produced under a still-valid policy and credential epoch?
Does the current tool contract interpret the result the same way?
Is the result safe to reuse for this operation, or was it only an observation?
Can the caller prove where the value came from?

Treating those as separate checks prevents a cache hit from silently becoming an authorization bypass.

Define the cache entry as evidence

Do not store only key -> value. Store the context that makes the value valid:

cache_key = hash(
  tool_name,
  normalized_arguments,
  principal_id,
  tenant_id,
  resource_scope,
  policy_version,
  credential_epoch,
  tool_contract_version
)

A useful entry also contains produced_at, expires_at, the producing run ID, a provider request ID, a sensitivity classification, a schema hash, and a revocation or invalidation sequence.

The key prevents accidental cross-context reuse. The metadata lets the read path reject entries that are technically fresh but no longer valid.

Separate observation from authority

The safest default is to cache observations, not permissions.

A cached “invoice total is 42” may be useful for display, but it should not be treated as proof that the current run may refund that invoice. The refund path should perform a current authorization check and, where necessary, a fresh resource read.

A practical classification is:

Class	Example	Reuse rule
OBSERVATION	list of recent logs	Reuse only within scope and freshness bounds
ADVISORY	model-generated summary	Never use as authorization evidence
DECISION_INPUT	account state used for a plan	Revalidate policy and resource version
EFFECT_CONFIRMATION	provider request ID	Reconcile by stable effect key, not TTL

This also protects against using a cache to “remember” that an external write succeeded. A timeout after dispatch is an UNKNOWN outcome. It needs provider reconciliation, not a cache hit that happens to contain a similar response.

Enforce the contract at read time

The read path should reconstruct the current context and reject mismatches before returning a hit.

def usable(entry, current, now):
    if entry is None or entry.expires_at <= now:
        return False

    same_context = (
        entry.principal_id == current.principal_id
        and entry.tenant_id == current.tenant_id
        and entry.resource_scope == current.resource_scope
    )
    same_versions = (
        entry.policy_version == current.policy_version
        and entry.credential_epoch == current.credential_epoch
        and entry.tool_contract_version == current.tool_contract_version
    )

    if not same_context or not same_versions:
        return False
    if current.use == "authorization" and entry.classification != "DECISION_INPUT":
        return False
    if current.use == "effect_confirmation":
        return False  # reconcile by effect key instead
    return True

The invariant is simple:

A cache hit is usable only when its validity context matches the current run and its classification permits the requested use.

Do not rely on the model to decide whether a cached value is safe. The model can request a cache read, but a deterministic layer should decide whether the read is allowed.

Make invalidation explicit

TTL is a fallback. Important changes should invalidate immediately:

policy version increment;
credential epoch increment;
tenant membership or resource ownership change;
tool schema or contract change;
manual incident response;
deletion or retention event;
provider reconciliation that contradicts the cached result.

An append-only invalidation record makes the behavior inspectable:

CREATE TABLE cache_invalidations (
  scope TEXT NOT NULL,
  version INTEGER NOT NULL,
  reason TEXT NOT NULL,
  created_at TIMESTAMP NOT NULL,
  PRIMARY KEY (scope, version)
);

Writers attach the current invalidation version to each entry. Readers reject entries written before the active version for the relevant scope.

That is safer than trying to find and delete every matching key during an incident. Deletion can be incomplete; a version boundary is easy to check and audit.

Prevent stampedes without widening authority

When many runs miss simultaneously, single-flight loading can help, but the lock must not blur security boundaries.

Use a lock key with the same principal, tenant, resource, and version dimensions as the cache key. Never let a privileged request populate a shared entry that an unprivileged request can read.

Bound the wait:

attempt a scoped cache read;
if missing, acquire a scoped single-flight lease;
recheck the cache after acquiring the lease;
load from the provider with current credentials;
validate the response schema and resource scope;
write the entry with its context and invalidation version;
release the lease.

If the provider call times out after dispatch, do not populate a success entry. Record UNKNOWN and reconcile with the provider using a stable request key.

Failure-injection checklist

Add these tests to CI or a staging drill:

change the tenant between write and read;
revoke the credential while the entry is within TTL;
increment policy version while the entry is fresh;
change the tool contract or result schema;
replay identical arguments under a different principal;
return an observation on an authorization path;
crash after provider dispatch but before cache write;
make the provider return an UNKNOWN timeout;
let two tenants contend for normalized arguments;
restore stale cache entries with current policy versions;
fill the cache with an oversized or sensitive result.

For every test, record the expected decision: HIT, MISS, REVALIDATE, DENY, or RECONCILE. A green suite should show not only that the request failed safely, but why.

What to monitor

Track cache behavior as a control-plane signal, not just a latency metric:

hit and miss rate by classification;
rejected hits by reason;
cross-context mismatch attempts;
policy and credential invalidations;
stale-entry reads after restore;
UNKNOWN outcomes awaiting reconciliation;
cache fill failures and provider request IDs;
sensitive-data eviction and retention events.

A rising hit rate can be a bug if rejection and revalidation data disappear. Keep enough evidence to answer which context produced a value and why the current run was allowed to use it.

If you need a managed place to run an always-on OpenClaw or browser-automation workload, managed OpenClaw hosting on Ampere is one option to evaluate. Hosting does not define cache authority, invalidation, credential scope, or reconciliation semantics; those contracts still belong in the application.

Final rule

A TTL answers “how old is this value?” It does not answer “who may trust it now?”

For agent systems, bind cache entries to identity, tenant, resource scope, policy version, credential epoch, and tool contract. Classify what the value is allowed to prove. Invalidate by version when authority changes. Reconcile external effects instead of treating cached responses as confirmation.

That turns caching from a speed optimization into an explicit, testable part of the agent's control plane.

Your AI Agent Upgrade Needs a Compatibility Gate, Not Just a Rollback

Zira — Thu, 20 Aug 2026 03:44:13 +0000

A rollback answers one question: How do I put the old binary back?

It does not answer the harder question: What happens to work, state, credentials, and queued effects that crossed the version boundary while the upgrade was running?

For an AI agent, that boundary is everywhere. A new worker may read an old state row, a queued tool call may use a changed schema, or a restarted process may replay an effect created by the previous release. A clean process rollback can still leave an incompatible state store and duplicate side effects.

This article presents a small compatibility gate you can run before and during an agent-runtime upgrade.

The contract to gate

Give every persisted object and outbound effect an explicit producer contract:

default_version = {
  state_schema: 7,
  tool_contracts: { browser_navigate: 3, repo_apply_patch: 2 },
  effect_protocol: 2,
  credential_epoch: 19
}

The exact format is not important. The important part is that the worker cannot silently assume that the current version is compatible. Persist the versions beside the run, task, queue item, and effect record.

A worker should accept work only when it can prove all of these conditions:

It can read the state schema and preserve fields it does not own.
Its tool-input and tool-output contracts match the queued request.
Its effect protocol understands the existing idempotency and reconciliation states.
Its credential epoch is still authorized for this run.
It can write evidence in a format the recovery tooling understands.

If any check is unknown, pause the item. Do not guess that an old payload is harmless.

Use expand, gate, migrate, contract

A safe rollout is usually four phases:

1. Expand

Deploy readers that understand both the old and new representation. Add new columns or fields without deleting old ones. New writers should continue producing the old representation until the compatibility gate is live.

2. Gate

Put the new worker in a canary mode. It may claim only runs whose contract passes the compatibility test. Keep a visible counter for rejected, unknown, and accepted items.

A useful gate result is more informative than a boolean:

def check_compatibility(item, worker):
    reasons = []
    if item.state_schema not in worker.readable_state_schemas:
        reasons.append("state-schema")
    if item.effect_protocol not in worker.effect_protocols:
        reasons.append("effect-protocol")
    if item.credential_epoch > worker.max_authorized_epoch:
        reasons.append("credential-epoch")
    return {"decision": "reject" if reasons else "accept", "reasons": reasons}

In production code, compare capabilities rather than requiring every version number to be identical. A worker that can safely read schema 6 and 7 should say so explicitly.

3. Migrate

Migrate durable state in resumable, idempotent batches. Record the source version, destination version, migration attempt, and checksum. If a migration stops halfway through, rerunning it should either complete the same transformation or report a deterministic conflict.

Never treat a successful database transaction as proof that an external effect completed. Keep execution state and delivery state separate. An upgrade can commit a dispatch-requested row immediately before the worker dies.

4. Contract

Only after the queue is drained or gated, the state is migrated, and reconciliation is complete should you remove the old reader and writer path. Keep the old path long enough to recover records created before the cutover.

What rollback must preserve

Before declaring a rollback successful, verify these invariants:

Every claimed run has one owner or an explicit expired lease.
Every tool call has a stable idempotency key.
Every outbound effect is CONFIRMED, NOT_SENT, or UNKNOWN, never silently absent.
Every UNKNOWN effect has a provider lookup or a human/operator decision path.
Credential revocation and policy changes apply to both old and new workers.
The evidence writer can still render the run history after the version change.

This is why a rollback should restore a compatible runtime, not merely an older container image.

A failure-injection drill

Run this in a staging environment with a disposable state store:

Queue a task using the old tool contract.
Claim it with the new worker, then kill the worker after the claim transaction.
Change the credential epoch before the worker restarts.
Deploy the old worker and attempt recovery.
Inject a timeout after an outbound provider accepts the request but before your ledger records the response.
Run reconciliation, then inspect whether the task is resumed, paused, or marked unknown.
Repeat with a partially migrated state row and a duplicate delivery attempt.

The expected result is not that everything finishes. The expected result is that every ambiguous transition becomes visible and no stale worker can perform an effect after losing authority.

Hosting is part of the recovery surface

If the agent must stay available during a migration or recovery window, a managed runtime can reduce the operational work of keeping the process online. For teams evaluating that option, managed OpenClaw hosting on Ampere is one hosting path to compare.

That does not provide the compatibility contract for you. You still need durable state, scoped credentials, migration tests, reconciliation, and a rollback plan that accounts for work already in flight.

The checklist

Before upgrading an agent runtime, ask:

Which state schemas can the new worker read and write?
Which queued tool contracts can it execute safely?
What is the effect protocol for requests that are UNKNOWN at the cutover?
How are credential and policy versions rechecked after restart?
Can the migration resume without duplicating a transformation?
Can the old runtime recover records created by the new runtime?
Which evidence proves the cutover was safe?

If you cannot answer those questions, you have a deployment plan and a rollback button, but not a compatibility gate.

The right upgrade test is not whether the new process started. It is whether old and new workers can share the boundary without losing authority, state, or effect evidence.

Your AI Agent's Health Check Lies Unless It Tests the Tool Path

Zira — Wed, 19 Aug 2026 23:42:59 +0000

A process can answer 200 OK while every useful action is failing.

That is the dangerous kind of healthy state for an AI agent. The supervisor sees a live process, the container stays running, and the dashboard is green. Meanwhile the agent cannot reach its model gateway, its tool credentials are expired, its browser profile is locked, or its queue consumer has lost its lease.

The fix is not to make one giant health endpoint that performs a real side effect. The fix is to model health as several explicit signals and test the path that matters without crossing the mutation boundary.

Separate liveness, readiness, and capability

Use at least three states:

Signal	Question	Safe check
Liveness	Is the process event loop responding?	Local response with a deadline
Readiness	Can this worker accept a new run?	Queue lease, config, and dependency checks
Capability	Can this run perform its required tool calls?	Non-mutating probes for the selected capability set

Liveness should be cheap. A liveness failure usually means the process needs a restart.

Readiness is an admission decision. A worker with a stale configuration version or no durable-state connection should stop receiving new work, even if its process is alive.

Capability is run-specific. A text-only task may need a model gateway and a database, while a browser task also needs a valid browser profile, network policy, and download storage. Do not report one global green light when only half the capabilities are usable.

Make the checks typed, not boolean

A boolean loses the information needed for recovery. Return a typed result with an owner and a next action:

{
  "component": "browser",
  "state": "DEGRADED",
  "checked_at": "2026-08-20T01:00:00Z",
  "config_version": "cfg-184",
  "credential_version": "cred-77",
  "evidence": {
    "profile_lock": "held-by-worker-12",
    "read_only_probe": "passed"
  },
  "admission": "reject_browser_runs",
  "recovery": "recycle_profile_after_lease_expiry"
}

Useful states are usually PASS, DEGRADED, FAIL, and UNKNOWN. UNKNOWN matters when a probe timed out or a dependency returned an ambiguous result. It should not silently become PASS, and it should not always trigger an aggressive restart.

A typed result also makes alerts actionable. “Agent unhealthy” is not an operator instruction. “Reject browser runs, keep text runs admitted, recycle the profile after the lease expires” is.

Probe dependencies without performing user mutations

A tool-path probe should be representative but reversible. Examples:

Model gateway: authenticate, request the smallest allowed metadata or token-count operation, then record latency and policy version.
Database: open a connection, verify schema version, and execute a read-only query with a deadline.
Object storage: verify access to the expected bucket and prefix, without uploading a test object unless the system has a cleanup contract.
Browser: launch a disposable profile, navigate only to an allowlisted diagnostic page, and verify the expected browser/CDP handshake.
MCP: list the server’s declared tools and compare them with the policy snapshot, without invoking a mutating tool.

Never use “send an email to test email.” A health check that creates an external side effect is an incident generator. If a provider cannot offer a safe probe, classify that capability as UNVERIFIED and rely on delivery receipts or a separate canary account with an explicit cleanup path.

Tie checks to the run's capability set

Before admission, compute the required capability set from the task, not from the worker's startup profile:

def admit(run, health):
    required = set(run.required_capabilities)
    missing = []

    for capability in required:
        result = health[capability]
        if result.state not in {"PASS", "DEGRADED"}:
            missing.append((capability, result.state))

    if missing:
        return {"decision": "REJECT", "missing": missing}

    return {
        "decision": "ADMIT",
        "health_snapshot": health.snapshot_id,
    }

Store the health snapshot ID with the run admission record. Otherwise an incident review cannot answer a basic question: did the worker admit the task before or after the dependency failed?

The snapshot is not permanent authorization. Recheck policy, credential version, and lease ownership at dispatch time. Health proves a narrow observation at a point in time; it does not grant permission to perform a later action.

Test the failure domains separately

A useful test matrix injects one fault at a time:

Keep the process responsive but block the model gateway. Liveness should pass; model capability should fail; new model runs should be rejected.
Expire the browser credential while the process remains alive. Browser capability should become degraded or failed; text-only work should remain unaffected.
Break the queue lease renewal. Readiness should fail after the lease deadline, and another worker should be able to claim new work.
Return a timeout from a tool dispatch. The result should be UNKNOWN, not an automatic retry that might duplicate a mutation.
Roll the configuration version while a run is waiting. The worker should recheck at dispatch and either refresh safely or reject the run.
Fill the diagnostic storage path. The probe should identify the capacity failure without turning every unrelated capability red.

For each test, record the expected admission decision, the alert, the recovery owner, and whether already-running work is allowed to finish. That last field prevents a readiness failure from becoming an unplanned cancellation storm.

Make recovery narrower than restart

A failed capability does not always justify restarting the whole agent. Prefer the smallest recovery domain:

Refresh a short-lived credential when only its version is invalid.
Recycle a disposable browser profile when the profile lock is stale.
Pause admission when the queue lease is lost.
Route around one unavailable model provider when policy permits.
Restart the process only when liveness or local invariants fail.

For always-on OpenClaw or browser-agent deployments, the hosting layer is part of this recovery contract. A managed runtime such as always-on OpenClaw hosting on Ampere can be a hosting option to evaluate when you need a persistent process and repeatable restart boundary, but it does not remove the need to define state persistence, credential scope, health probes, and recovery policy yourself.

What to put on the dashboard

Show separate panels for:

process liveness and restart count;
readiness and admission rejections;
capability state by dependency and task type;
probe latency and timeout rate;
configuration and credential versions;
unknown outcomes awaiting reconciliation;
recovery actions and their completion status.

The goal is not a prettier green dashboard. The goal is to make “alive but unable to do useful work” impossible to confuse with “ready to accept this run.”

If you build agents, add one tool-path probe this week. Then inject one dependency failure while the process stays alive. If the system cannot reject only the affected work, preserve the evidence, and recover the smallest possible domain, the health check is measuring uptime, not health.