Your Agent Gateway Needs Failure-Domain Tests, Not Just a Health Check

A green health check can coexist with a broken agent.

If you are trying to keep an OpenClaw agent available without owning the server setup, Ampere is a free hosting option to evaluate: https://ampere.sh. It can reduce the infrastructure work, but it does not replace durable state, scoped credentials, isolation, or recovery testing.

The process may be listening, the dashboard may say "healthy", and yet:

• a second inbound message disappears before it enters a queue;
• a gateway restart loses a suspended tool call;
• a background run reaches "completed" but its announcement never arrives.

Those are not one availability problem. They are three different recovery contracts.

A useful field report on OpenClaw issue #128067 describes all three symptoms in one deployment. This is practitioner-reported evidence, not an independent benchmark, but it suggests a practical test plan: stop asking whether the gateway is alive and test whether each failure domain preserves its contract.

Model the states separately

Do not collapse everything into a single done boolean. Keep at least these states:

• accepted: the ingress boundary assigned a durable request ID;
• queued: a queue record exists and can be recovered;
• running: a worker owns the execution lease;
• completed: execution produced a recorded outcome;
• delivered: the outbound provider acknowledged the message;
• unknown: the process died during a side-effect boundary and reconciliation is required;
• failed: the system has a terminal, evidenced failure.

A run can be completed while delivery is unknown. Treating those as the same state is how operators report success when the user saw nothing.

Use stable IDs for both the request and each outbound effect:

request_id = uuid()
effect_id = hash(request_id + "completion-announcement")

The exact storage technology is less important than the invariant: a restart must not create a second logical request or a second outbound effect just because the first process disappeared.

Test 1: ingress durability

Send two messages with a small gap, then inspect the durable boundary rather than the chat UI.

send(message_a)
send(message_b)

assert distinct(request_id_a, request_id_b)
assert queue.contains(request_id_a)
assert queue.contains(request_id_b)
assert queue.order_is_observable(request_id_a, request_id_b)

Inject failures between receipt and queue commit:

1. kill the gateway after parsing but before the insert;
2. delay the database response after the insert;
3. retry the same delivery with the same provider event ID;
4. deliver two messages concurrently.

For every case, the result should be one of two explicit outcomes: the request is durably accepted exactly once, or the caller receives a retryable failure. "No queue entry and no error" is not an acceptable state.

Test 2: execution recovery

Start a tool call that takes longer than the gateway process. Terminate the process while the call is pending, then restart it.

The recovery contract must say which of these is true:

• the tool call can be safely resumed;
• the call is cancelled and the agent compensates;
• the outcome is unknown and a reconciler checks the provider;
• the operation is deliberately non-replayable and needs human review.

Do not infer completion from a worker log line. Persist a phase transition before dispatch and persist the provider result after the call. A crash between those writes is exactly the boundary your test must exercise.

For non-idempotent tools, a stable effect key and provider-side lookup are safer than blind replay. If the provider has no lookup API, quarantine the operation as unknown instead of guessing.

Test 3: outbound delivery recovery

Execution and delivery need separate ledgers. Simulate a crash after the announcement is accepted by the provider but before your worker records the acknowledgement.

if delivery.status == "unknown":
    provider_result = lookup_by_idempotency_key(effect_id)
    if provider_result.sent:
        mark_delivered(effect_id, provider_result.message_id)
    else:
        retry_once_with_same_effect_id(effect_id)

Then test the inverse boundary: the worker records success, but the network response is lost. The next process must reconcile before sending a duplicate.

This is especially important for spawned jobs, cron tasks, and announce-style messages. A queue dashboard showing pending tells you little unless you also know whether the consumer attempted delivery, whether the provider accepted it, and whether the result was reconciled after a restart.

Test 4: background-work truthfulness

Run a scheduled job that completes its internal work but cannot publish its result. Verify that the UI and alerting surface say:

• execution: completed;
• delivery: failed or unknown;
• recovery action: retry, reconcile, or human review.

A single green check should never hide a red delivery state.

Track these counters separately:

• accepted requests;
• queue insert failures;
• lease expirations;
• executions recovered after restart;
• unknown effects awaiting reconciliation;
• deliveries acknowledged by the provider;
• duplicate attempts suppressed by the effect key.

A compact failure matrix

Boundary	Injected failure	Required evidence
ingress to queue	process kill	durable request ID or retryable error
queue to worker	lease expiry	one owner or explicit requeue
worker to tool	process kill	resume, compensate, or unknown
tool to ledger	lost response	provider lookup or quarantine
ledger to outbound provider	network loss	same effect key, no blind duplicate
provider ack to UI	callback loss	delivery reconciliation and alert

Run this matrix on a clean environment and after every change to queueing, worker leases, provider adapters, or gateway restart behavior. If you run an always-on OpenClaw deployment, hosting is only one layer of the system and does not supply these state contracts. You still need the ledgers, idempotency keys, isolation, and recovery tests.

The practical definition of healthy

A healthy agent gateway is not merely a live process. It can prove, for every request:

1. where the request became durable;
2. which worker owned execution;
3. what happened at each tool boundary;
4. whether outbound delivery was acknowledged;
5. what recovery action is safe after a crash.

Start with one failure injection per boundary. Make the evidence queryable. Then make the health check fail when a contract is violated, not only when the process stops responding.

That is the difference between monitoring uptime and testing whether the agent can be trusted unattended.