Your AI Agent's Dead-Letter Queue Needs a Recovery Contract

A dead-letter queue is not a recovery plan. It is only a place to put work that the normal path could not finish.

For an AI agent, that distinction matters because a failed item may represent:

• a pure computation that is safe to replay;
• an external side effect whose outcome is unknown;
• a policy violation that must never be retried automatically; or
• a poisoned input that will fail forever under the same conditions.

If every dead-letter item gets a generic “retry” button, operators eventually create duplicate side effects, retry expired credentials, or bury real incidents under a growing pile of unclassified work.

This article describes a small recovery contract you can implement around a queue, worker, and operator console.

1. Store a decision-ready dead-letter record

Do not copy only the original payload into a dead-letter table. Store the facts needed to decide what happens next.

A minimal record looks like this:

global_id: 01J...
run_id: run_4821
effect_id: effect_913
attempt: 4
failure_class: UNKNOWN
status: DEAD_LETTERED
first_failed_at: 2026-08-18T07:00:00Z
last_failed_at: 2026-08-18T07:12:00Z
lease_fencing_token: 19
credential_version: github-token-v7
policy_version: policy-42
payload_digest: sha256:...
next_action: NEEDS_RECONCILIATION

The payload digest lets you detect accidental mutation. The run and effect IDs let you join the item to execution and outbound-delivery evidence. The fencing token and credential version tell you whether an old worker is trying to resume work after ownership or authorization changed.

A dead-letter row should be immutable apart from explicitly audited fields such as status, operator decision, and reconciliation result.

2. Classify the failure before choosing a retry

Use an explicit failure taxonomy instead of deriving retry behavior from an exception string.

Class	Meaning	Automatic action
TRANSIENT	The operation did not start or the provider gave a retryable error	Retry within a bounded budget
POISONED_INPUT	The same validated input will fail repeatedly	Quarantine and fix the input
POLICY_DENIED	Current policy forbids the action	Do not retry; request approval or change policy
CREDENTIAL_EXPIRED	The credential lease or version is no longer valid	Re-authorize, then re-evaluate
UNKNOWN	The process stopped during the effect window	Reconcile provider state before replay
NON_RETRYABLE	A retry would be unsafe or meaningless	Require an explicit operator decision

The key rule is that UNKNOWN is not a failure verdict. If the worker timed out after sending a request, the queue cannot safely infer that nothing happened.

This is the same boundary that makes work-path health checks for AI agents more useful than a process heartbeat: a live process does not prove a recoverable operation.

3. Separate replay from reconciliation

Give operators two different actions:

• Reconcile: ask the external system whether the effect happened, using the stable effect ID or provider request ID.
• Replay: create a new attempt only after the system has established that replay is safe.

For example, a browser automation step that may have submitted a form should not be replayed merely because the browser worker disconnected. First check the application, confirmation endpoint, or provider event log. If the provider cannot answer, keep the item in UNKNOWN rather than converting uncertainty into a duplicate submission.

A useful state machine is:

graph TD
  A[DEAD_LETTERED] --> B{failure class}
  B -->|TRANSIENT| C[RETRY_ELIGIBLE]
  B -->|UNKNOWN| D[RECONCILIATION_REQUIRED]
  B -->|POLICY_DENIED| E[APPROVAL_REQUIRED]
  B -->|POISONED_INPUT| F[INPUT_REPAIR_REQUIRED]
  D --> G{effect confirmed?}
  G -->|yes| H[COMPLETE]
  G -->|no, safe| C
  G -->|no answer| I[UNKNOWN_HOLD]
  C --> J[NEW_ATTEMPT]

If DEV renders this as text rather than a diagram, that is intentional: the state transitions are the contract, not the visualization.

4. Make recovery idempotent

Every recovery decision needs its own idempotency key. A double-click, browser refresh, or two operators working at once must not create two replay attempts.

For example:

def start_recovery(item_id, decision, operator_id):
    key = f"recover:{item_id}:{decision.version}"
    with transaction():
        decision_row = insert_decision_if_absent(
            key=key,
            item_id=item_id,
            decision=decision.kind,
            operator_id=operator_id,
        )
        if not decision_row.created:
            return decision_row.result

        assert current_status(item_id) == "DEAD_LETTERED"
        assert decision_matches_current_versions(item_id)
        return create_recovery_attempt(item_id, key)

The transaction should atomically record the decision and create the recovery attempt. If those operations cannot be atomic, use a durable outbox and make the consumer idempotent too.

5. Re-check authority at recovery time

A dead-letter item may sit for hours. Never assume that the original authorization is still valid.

Before replaying, re-check:

1. tenant and resource scope;
2. current policy version;
3. credential lease and credential version;
4. tool capability and destination;
5. whether the original operator or workflow is still allowed to act.

A hosting provider can keep the worker running, but it does not decide whether an old dead-letter item is still authorized or whether an unknown external effect is safe to repeat. If the operational problem is keeping an OpenClaw runtime available while you handle these recovery states, managed OpenClaw hosting on Ampere is one option to evaluate. It does not replace the queue contract, authorization checks, or reconciliation logic.

6. Test the recovery path with failure injection

A recovery contract is only real if the team can exercise it. Add tests that stop the worker at each dangerous boundary:

• after the provider accepts a request but before the response is persisted;
• after the dead-letter row is written but before the alert is delivered;
• while a second operator submits the same recovery decision;
• after the credential lease expires but before a new attempt starts;
• after policy changes but before a queued retry is dispatched;
• after the external provider returns no answer during reconciliation;
• after the recovery worker loses its durable state connection.

For each test, assert the final evidence, not just the HTTP response:

• exactly one recovery decision exists for the idempotency key;
• no stale fencing token can dispatch work;
• UNKNOWN remains UNKNOWN when the provider cannot establish an outcome;
• a confirmed effect is not replayed;
• a policy or credential change blocks the old attempt;
• an operator can see why the item is waiting and what evidence is missing.

A practical operator checklist

Before pressing “replay,” ask:

• What exactly failed: execution, delivery, authorization, or evidence persistence?
• Could the external effect already have happened?
• Which stable identifier can prove that outcome?
• Is the input unchanged and still valid?
• Are the policy and credential versions current?
• Is this replay within the retry and cost budget?
• What happens if the provider stays unavailable?
• Can another operator or worker submit the same decision?

If the console cannot answer those questions, it is not a recovery console yet. It is a manual retry endpoint with better typography.

The useful design goal is not “make every dead-letter item retryable.” It is to make every item classifiable, auditable, and safe to resolve. That gives operators a bounded path from failure to evidence-backed completion, even when the agent, worker, or provider disappears at the worst possible moment.