DEV Community

Zira
Zira

Posted on

Your Agent Queue Needs Backpressure, Not Just More Workers

#agents #devops #testing #openclaw

Long-running agents rarely fail because a single request is too large. They fail when work arrives faster than the runtime can safely execute it.

A webhook burst, a retry storm, or a slow model provider can turn an apparently healthy agent into a pile of waiting tool calls. Adding workers may reduce latency for a while, but it can also multiply browser sessions, spend, rate-limit pressure, and duplicate side effects.

The missing control is backpressure: a deliberate decision to stop accepting work before the queue becomes unbounded.

Define the queue contract first

Before changing concurrency, write down four numbers:

  • max_pending: work accepted but not started
  • max_running: work currently executing
  • lease_ms: how long a worker may hold a job before renewal is required
  • retry_budget: how many retries a job receives before becoming blocked

A useful invariant is:

gross capacity = max_running * average safe throughput

The queue should reject or defer new work when pending + running crosses a limit derived from that capacity. Do not use memory usage alone as the limit. A browser session, an MCP server, and a model call can each have different resource costs.

Make admission explicit

Here is a small admission check. The important part is that rejection is a durable result, not an exception that causes the caller to retry immediately.

def admit(stats, job):
    if stats.pending + stats.running >= stats.max_in_flight:
        return {"accepted": False, "reason": "overloaded", "retry_after_s": 30}

    if job.cost_class == "browser" and stats.browser_running >= stats.browser_limit:
        return {"accepted": False, "reason": "browser_limit", "retry_after_s": 15}

    return {"accepted": True}
Enter fullscreen mode Exit fullscreen mode

Persist the decision with the job ID and an idempotency key. A caller that receives overloaded should back off or show a deferred state. It should not create a new job with a new ID just because the first admission was refused.

Separate pending, running, and unknown

A queue dashboard with only pending and done hides the most dangerous state: work whose outcome is unknown.

Use at least these states:

  1. accepted: admission succeeded and the job is durable
  2. running: a worker owns a current lease
  3. succeeded or failed: the outcome was confirmed
  4. unknown: the worker or provider disappeared after a side effect may have happened
  5. blocked: retry budget or policy requires human or automated reconciliation

Do not move unknown straight back to pending. First query the provider or inspect your side-effect ledger. This is especially important for browser clicks, outbound messages, payments, and deployment commands.

My earlier interruption-ledger pattern for crashed tool calls is useful here: queue pressure tells you when to stop admitting work, while the ledger tells you how to reconcile work already dispatched.

Use leases, not permanent ownership

A worker can die while holding a job. A permanent running flag then becomes a silent leak.

A lease record should include:

job_id=job_123
worker_id=worker_7
lease_expires_at=2026-08-11T18:30:00Z
attempt=2
idempotency_key=repo-42:issue-91:comment
Enter fullscreen mode Exit fullscreen mode

Renew the lease only while the worker is alive and making progress. On expiry, put the job into unknown first. Recovery code can decide whether it is safe to retry, rather than assuming that an expired lease means no side effect occurred.

Test the failure modes deliberately

A useful local test does not need a production-scale load generator. Start with a queue of 20 jobs and inject one failure at a time:

  • pause admissions after job 5
  • kill a worker after it records running but before confirmation
  • make the model provider return 429 for 60 seconds
  • make browser startup take longer than the lease
  • deliver the same webhook twice
  • restart the queue process while jobs are pending

For each case, verify:

  • the queue stops accepting work at the configured boundary
  • rejected jobs are not silently duplicated
  • expired leases become unknown, not automatically succeeded
  • retries preserve the idempotency key
  • operators can see pending, running, unknown, and blocked counts separately

A green test is not enough if it never forces the guard to return false. Keep one test that proves admission is refused and one that proves recovery does not replay a potentially completed side effect.

Hosting is part of the queue design

If an OpenClaw or browser-automation worker must stay online, treat the host as part of the control plane: persistent state, restart behavior, logs, backups, and a safe rebuild path matter more than a process that merely stays running. A managed option such as always-on OpenClaw hosting on Ampere is worth evaluating only against that checklist, not as a substitute for queue semantics.

The practical acceptance test is simple: stop the worker, restart it on a clean instance, restore the queue state, and prove that pending work resumes while unknown work remains explicitly reconcilable.

The rule of thumb

Add workers only after you can answer these questions:

  • What is the admission limit for each expensive resource?
  • What does the caller observe when the limit is reached?
  • How do you distinguish no execution from unknown execution?
  • Which operations are safe to retry?
  • Can you rebuild the runtime without losing the queue contract?

More concurrency is an optimization. Backpressure, leases, and reconciliation are the safety mechanism. Build those first, then measure whether another worker actually improves the system.

Top comments (0)