How async job processing moves long-running agent work out of HTTP requests
Long-running agent work looks fine right up until it hits real traffic. Then requests start hanging, workers stay busy too long, retries get messy, and users have no clear idea whether anything is still happening. The fix is to move that work out of the request cycle: validate input, enqueue it in a background job queue, return 202 Accepted with a job_id, and let workers finish the task outside the synchronous path.
A practical flow is POST /jobs → queue → worker → status store. Clients poll GET /jobs/{id} or subscribe over SSE or WebSockets for progress, then fetch /jobs/{id}/result. But enqueueing is the easy part. Correctness is where most of the work lives: use idempotency keys, worker leases with renewal, retries with caps, dead-letter queues, cancellation flags, and OpenTelemetry traces. At Imversion Technologies Pvt Ltd, that bias toward clarity over complexity helps teams make HTTP request offloading reliable instead of merely asynchronous.
Key Takeaways for async job processing
- Synchronous requests break down once long running tasks start calling agents, tools, or large pipelines -- web workers get pinned, timeouts rise, and users get no clean recovery path. Use HTTP request offloading: accept work, enqueue it in a background job queue, and return
202 Acceptedwith ajob_id. - Correctness is the hard part. Worker leases, visibility timeouts, and idempotency keys stop duplicate execution and let crashed jobs retry safely.
- A good status API -- like
GET /jobs/{id}andGET /jobs/{id}/result-- turns async job processing into something users can follow instead of guess about. - SSE or WebSockets improve UX with live progress, but they do not replace durable job state in the database or queue.
- Keep retries bounded, route poison messages to a dead-letter queue, support cancellation, and instrument queues, workers, and jobs with OpenTelemetry. Reliable systems matter most.
Table of Contents
- How async job processing moves long-running agent work out of HTTP requests
- Key Takeaways for async job processing
- Why synchronous agent requests fail and the architecture that replaces them
- Design the background job queue with idempotency, small messages, and worker leases for async job processing
- Build a job status API and stream progress with WebSockets or SSE for async job processing
- Handle retries, dead-letter queues, cancellation, and observability before shipping
- A practical async job processing blueprint for agent APIs
-
Frequently Asked Questions
- What is the biggest mistake teams make when implementing async job processing?
- How does async job processing affect API rate limiting and fairness?
- Why should I separate user-facing status from internal worker events?
- How should I decide between polling, SSE, and WebSockets?
- What data should I retain for completed async job processing runs?
Why synchronous agent requests fail and the architecture that replaces them
The trouble starts when the web request becomes the home for work it was never meant to carry. Long-running agent tasks might behave in development, where traffic is light and dependencies are cooperative. Production changes the picture fast. Agent calls fan out, tools slow down, and the path begins to time out in ways that are hard to recover from cleanly.
Most gateways, load balancers, and app servers impose request time limits. Even before a hard timeout, the damage starts: blocked web workers, rising queue depth at the app tier, weak retry behavior, and no durable record of what was in flight. A client retries. The server may run the same work twice. Or lose it if the process crashes after returning 202 Accepted but before a real handoff.
Returning
202 Acceptedis not the architecture. Durable state plus worker handoff is.
The replacement flow
The replacement is straightforward in shape, but it has to be strict about handoff and ownership:
client → API → durable queue → workers → result store → job status API
POST /jobs should validate input, create a job record, enqueue a small message, and return 202 Accepted with a job_id, /jobs/{id}, and /jobs/{id}/result. Use a durable queue such as SQS, RabbitMQ, Redis Streams, or Kafka. Keep messages lean: store large payloads outside the queue and pass references.
From there, workers claim jobs with a lease or visibility timeout, renew the lease while processing, checkpoint progress, and mark completion explicitly. If a worker dies, the lease expires and another worker can retry. That is the real goal: not background work by itself, but background work that recovers correctly.
What the async design must include
This is where many implementations get weaker than they look. A background job queue needs operational guardrails: status APIs, SSE or WebSockets for progress, retries with idempotency keys, dead-letter queues after bounded retry counts, cancellation via POST /jobs/{id}/cancel, and observability for queue lag, lease age, retry count, failure reason, and traces across API and workers.
| Dimension | Synchronous in request | Async job processing |
|---|---|---|
| Timeout risk | High for long running tasks | Low; request returns fast |
| Scalability | Tied to API worker count | Workers scale independently |
| User experience | Spinner, then failure | Polling or SSE/WebSocket progress |
| Recovery | Weak; duplicates and lost state | Retries, leases, DLQ, status history |
One caveat. Start simple. Use a FIFO queue, explicit job states, small messages, and bounded retries. Clarity beats complexity.
Design the background job queue with idempotency, small messages, and worker leases for async job processing
If ownership is vague, retries become dangerous. If messages are too heavy, retries become expensive. If job state lives only in the queue, debugging gets painful. That is why a reliable background job queue is built around correct ownership, safe retries, and clear job state, not clever routing.
Do not treat the queue message as the job itself. Keep a durable job record in a database with fields like job_id, status, idempotency_key, attempt_count, payload_ref, lease_owner, lease_expires_at, and checkpoint. Then put a small message on SQS, RabbitMQ, or Redis Streams that contains only the identifiers needed to fetch and process that record.
Keep queue messages small. If an agent run needs a large prompt, tool context, or uploaded file, store it in object storage or a database row and pass a reference such as payload_ref. That keeps retries cheap and avoids queue size limits, but it adds one more storage lookup during processing.
Separate job records from queue messages
Use the API flow POST /jobs → write job row → enqueue {job_id, idempotency_key} → return 202 Accepted. If the client retries the same request, the idempotency_key should return the existing job instead of creating a duplicate. For example, if a network timeout happens after enqueueing, the client can safely retry and get back the original job_id.
Start with one queue and one worker type unless you already know some jobs need isolation. Add FIFO ordering or priority queues only when there is a clear requirement, because they improve control but also increase operational complexity and debugging cost.
For long-running tasks, split one giant agent run into resumable steps: plan, fetch tools, generate output, persist result. Store progress with checkpointing so a retry resumes from the last safe step instead of replaying everything.
Use leases, not trust
Once jobs can take time, trust is not a control mechanism. Workers should claim jobs with a visibility timeout or lease. In SQS, an in-flight message becomes visible again when the timeout expires unless the worker deletes it. RabbitMQ acknowledgments and Redis Streams consumer groups support the same ownership pattern, even if the mechanics differ.
Workers should renew the lease periodically, update heartbeat fields, and mark completion explicitly.
If a worker crashes, the lease expires. Another worker can pick up the job and continue from the checkpoint. If retries keep failing, move the message to a dead-letter queue, expose that status on /jobs/{id}, and allow cancellation to set a terminal state that workers check before each step. Emit retry counts, lease age, queue depth, and step timings so silent failures stay visible.
Build a job status API and stream progress with WebSockets or SSE for async job processing
A client can tolerate waiting. What it cannot tolerate is uncertainty. If work moves out of the original request, the system needs a clean contract for status, errors, and final results.
For async job processing, the client contract should be simple: accept work quickly, return a durable job_id, and expose one clear place to check progress, errors, and final results without holding the original HTTP request open.
POST /jobs should validate input, create the job record, enqueue work on the background job queue, and return 202 Accepted with job_id, status_url, and result_url. If clients may retry, support an idempotency key so duplicate submits do not create duplicate long-running tasks.
GET /jobs/{id} is the center of the job status API. Keep the state machine boring: queued, running, succeeded, failed, canceled. Add retrying only if clients truly need to understand backoff behavior. Return metadata that helps both users and operators: created_at, started_at, updated_at, finished_at, attempt, max_attempts, progress_percent, current_step, error, cancel_requested, and links to result or events.
A response can look like this: {"id":"job_123","state":"running","progress_percent":60,"current_step":"tool_call","attempt":2,"created_at":"...","started_at":"...","updated_at":"..."}.
GET /jobs/{id}/result should return the final artifact only after success. Before that, return a clear pending response or direct clients back to status.
Start with polling. For most HTTP request offloading flows, polling every few seconds is enough. It is simpler, easier to reason about during retries, and avoids adding another connection model too early. Use Server-Sent Events for one-way live progress such as step updates, retry notices, or partial logs. Use WebSocket only when clients must both send and receive in real time, such as interactive cancellation, token streaming, or multi-step agent traces. The tradeoff is operational complexity: streaming feels nicer, but it adds connection lifecycle, backpressure, and reconnect behavior that a plain status API can often avoid.
Handle retries, dead-letter queues, cancellation, and observability before shipping
This is the part teams tend to postpone because the happy path already works. That delay is expensive. Long running tasks on a background job queue need failure controls before release, or silent loss, duplicate work, and stuck jobs will show up right after HTTP request offloading looks “done.”
Retry policy comes first. Use exponential backoff with jitter for transient failures: rate limits, network timeouts, short-lived downstream outages, lease conflicts. Jitter matters because synchronized retries can stampede the same dependency. But not every failure deserves another attempt. Validation errors, missing required inputs, unsupported tool calls, and deterministic parsing failures are usually permanent. Classify them early. Then stop retrying and move the job to a dead-letter queue after a bounded retry count or age threshold. Unlimited retries hide bad states instead of fixing them.
Because understanding why a job failed is essential, store failure class, retry count, last error, and next retry time in the job record and expose the summary through the job status API.
Cancellation needs honesty. POST /jobs/{id}/cancel should usually mark cancel_requested, not instantly canceled. Workers must check a cancellation token at safe checkpoints -- between tool calls, batch steps, or lease renewals -- and exit cooperatively.
Monitor the system like an operator, not a demo builder:
- queue depth
- age of oldest job
- lease expiry count
- retry rate
- success and failure rate
- dead-letter queue volume
- end-to-end latency
And trace across API and worker boundaries with OpenTelemetry and trace propagation. If a job disappears, the trace should show exactly where.
A practical async job processing blueprint for agent APIs
What usually breaks first is not the queue. It is the lack of a disciplined flow around it. The smallest production-aware design stays simple on purpose: accept fast, hand work to a durable queue, and make job state visible from start to finish. That is the core of async job processing. Teams often overbuild orchestration before they have stable job semantics. Wrong move.
Minimal end-to-end flow
A good baseline looks like this:
-
POST /jobsvalidates input, auth, and an idempotency key. - The API writes a
jobsrow withstatus=queued, input reference, attempt count0, and timestamps. - It publishes a small queue message like
{job_id, tenant_id, attempt}to a background job queue. - It returns
202 Acceptedwithjob_id,/jobs/{id}, and/jobs/{id}/result.
Then the worker queue architecture takes over. A worker pulls the message, acquires a lease -- SQS visibility timeout, RabbitMQ ack window, or Redis Streams claim flow -- and marks the job running. During long running tasks, it renews the lease, persists progress such as step, percent, and last_heartbeat, and emits progress over SSE or WebSockets if the client is listening.
Because understanding why is essential, every persisted state change should explain ownership and recovery: who has the job, when the lease expires, and whether retry is safe.
Status, retries, and terminal failure
Expose two plain endpoints: GET /jobs/{id} for status and GET /jobs/{id}/result for final output. If work fails transiently, increment attempts, record the error class, and requeue with backoff. If retries are exhausted, move the message to a DLQ and mark the job failed.
Cancellation needs a real path. POST /jobs/{id}/cancel should set cancel_requested=true; workers must check that flag between tool calls and before expensive steps.
Prototype means “it runs.” Production means “it recovers.”
Stack choices and fit
- SQS + Lambda: low ops, strong fit for bursty HTTP request offloading, but long jobs and lease renewal need care.
- RabbitMQ + container workers: better for custom routing, steady workloads, and tighter worker control.
- Redis Streams + app workers: fast and simple for small teams, but operational discipline matters more as scale and durability needs rise.
Checklist for readiness: idempotency keys, lease renewal, status endpoint, retry policy, DLQ, cancellation checks, and OpenTelemetry traces tied to job_id. Reliable systems matter most.
Frequently Asked Questions
What is the biggest mistake teams make when implementing async job processing?
The biggest mistake is treating the queue as the system of record. Reliable async job processing requires durable job state outside the queue so status, retries, cancellations, and recovery decisions survive worker crashes, message redelivery, and temporary broker outages.
How does async job processing affect API rate limiting and fairness?
Async job processing changes rate limiting from request time to work admission time. The API can accept requests quickly while enforcing tenant quotas, queue priorities, concurrency caps, and per-customer backpressure so one noisy client does not consume all worker capacity.
Why should I separate user-facing status from internal worker events?
User-facing status should stay stable and simple even when internal execution is complex. A clean external model reduces client coupling, while detailed internal events can still power debugging, audits, and analytics without forcing every consumer to understand leases, retries, and step-level transitions.
How should I decide between polling, SSE, and WebSockets?
Polling is best when updates are infrequent and simplicity matters most. SSE is a strong default for one-way progress streams because it is lighter than WebSockets and easier to reconnect. WebSockets are worth the added complexity only when clients need real-time bidirectional interaction.
What data should I retain for completed async job processing runs?
Keep enough data to support support, compliance, and performance analysis: final status, timing fields, attempt history, error summaries, input and result references, cancellation signals, and trace identifiers. Retention for raw payloads should be shorter and policy-driven because storage cost and privacy risk rise quickly.
Top comments (0)