Your Agent Queue Is Full. What Happens to the Work?

An agent can be “up” while its queue is quietly turning into a reliability incident.

A request may be accepted by an API, placed in a broker, claimed by a worker, sent to a tool, and still have no durable record of whether the side effect happened. If the process crashes between those steps, a retry can duplicate work or silently lose it.

This post describes a small control plane for backpressure and ambiguous delivery: explicit states, bounded queues, leases, and a reconciliation loop.

Start with two different facts

Do not use one status field for both questions:

1. Delivery: did the control plane durably accept the work?
2. Execution: did the worker complete the requested operation?

A minimal record can keep them separate:

Field	Example values
delivery_state	REJECTED, ACCEPTED, EXPIRED
execution_state	PENDING, RUNNING, SUCCEEDED, FAILED, UNKNOWN
request_key	client-generated idempotency key
attempt	monotonically increasing integer
lease_until	timestamp owned by the worker
payload_hash	normalized request digest

UNKNOWN matters. A timeout after a browser click, payment request, or deployment trigger does not prove that the side effect was absent.

Reject before the queue becomes a database

Backpressure should be a deliberate policy, not an accidental memory limit. At admission time, check:

• queue depth and oldest-item age;
• per-tenant and per-agent limits;
• estimated work cost, including tool calls and browser time;
• deadline or expiry time;
• whether the request key already exists;
• whether the downstream dependency is accepting new work.

For example:

if queue_depth >= hard_limit:
    return 429, {"reason": "queue_full", "retry_after": retry_window}

if oldest_age > max_age:
    return 503, {"reason": "stale_backlog"}

if duplicate_request_key:
    return existing_delivery_and_execution_state()
~~~

A soft limit can slow admission or reduce concurrency. A hard limit should fail quickly and predictably. Never let an agent infer “accepted” from an HTTP connection that was merely opened.

## Claim work with an expiring lease

A worker should atomically claim one item and record a lease. The lease prevents two healthy workers from processing the same item at the same time, but it is not proof that a crashed worker did nothing.

The claim transaction should record worker identity, runtime version, lease expiry, attempt number, a trace or run ID, and the normalized payload hash.

On restart, do not blindly relaunch every expired lease. Reconcile against the side effect first. For an external API, query by request key if supported. For a browser action, inspect a durable checkpoint or a provider-side result. If neither exists, classify the outcome as UNKNOWN and route it to a human or an operation-specific recovery handler.

## Keep an outbox boundary around side effects

The dangerous window is usually:

1. write intent;
2. send the side effect;
3. receive a response;
4. commit the result.

A crash can happen between any two lines. Persist the intent before dispatch, and make the request key stable across retries:

~~~sql
CREATE TABLE dispatch_intent (
  request_key TEXT PRIMARY KEY,
  payload_hash TEXT NOT NULL,
  operation TEXT NOT NULL,
  state TEXT NOT NULL,
  attempt INTEGER NOT NULL DEFAULT 0,
  created_at TEXT NOT NULL,
  updated_at TEXT NOT NULL
);
~~~

The executor should accept only the stored payload for that key. If a retry presents a different payload hash, stop rather than turning one idempotency key into two meanings.

## Test the failure points, not just the happy path

A useful fixture matrix includes:

- crash before the queue insert;
- crash after insert but before acknowledgement;
- worker death before dispatch;
- timeout after dispatch but before the response;
- response received but result journal not committed;
- lease expiry while the original worker is still alive;
- queue growth beyond soft and hard limits;
- duplicate request key with the same and different payload hashes;
- restart during reconciliation;
- downstream recovery while stale work is still queued.

For every fixture, assert both delivery and execution state. Also assert that a restart does not create a second side effect unless the operation explicitly supports safe replay.

## A 10-minute operator drill

1. Set worker concurrency to one.
2. Enqueue a read-only task and a deliberately slow task.
3. Fill the queue past the soft limit and confirm admission changes.
4. Kill the worker before dispatch, then after dispatch but before response.
5. Restart it and capture the reconciliation decision.
6. Submit the same request key twice, then once with a changed payload.
7. Record queue depth, oldest age, lease expiry, attempt, and final state.

If the dashboard only says “running,” the drill has not tested enough. Operators need to see accepted, claimed, dispatched, succeeded, failed, expired, and unknown work separately.


## Practical checklist

- [ ] Delivery and execution are separate state machines.
- [ ] Admission has soft, hard, per-tenant, and age limits.
- [ ] Workers use expiring leases and stable request keys.
- [ ] Intent is durable before a side effect is sent.
- [ ] Payload hashes prevent key reuse with different work.
- [ ] UNKNOWN is handled explicitly after ambiguous timeouts.
- [ ] Restart reconciliation is tested with injected crashes.
- [ ] Queue depth and oldest age are observable.
- [ ] The operator can explain what happened to every accepted request.

A full queue is not automatically a failure. An unclassified queue is. Backpressure protects the runtime; durable intent and reconciliation protect the work.