Short answer: For a SaaS app draining customer-support background jobs into rate-limited HTTP workers, use a region-local queue with delayed jobs, bounded retries, a dead-letter queue (DLQ), and a cron trigger that only enqueues work; make every public webhook endpoint idempotent before increasing concurrency. The queue product matters less than that delivery contract. This design accepts duplicate delivery as normal, treats HTTP 429 as backpressure rather than a generic failure, and stops poison jobs from consuming the worker pool forever. It also gives operators a clean control surface: lower dispatch concurrency, pause one tenant, replay a reviewed DLQ entry, or roll back the worker without losing the original intent. The catch is operational ownership. A managed queue is a sensible default for a small platform team because the on-call surface stays narrow, but it isn't suitable when policy requires broker-level control, when region availability doesn't match data residency, or when the workload needs strict global ordering. In those cases, keep a self-hosted broker or a database-backed dispatcher and budget explicitly for upgrades, storage pressure, and recovery drills.
What failure signal should control the worker pool?
Start with the downstream support API's rate limit, not CPU utilization in the queue consumer. HTTP 429 means the client has sent too many requests in a period; the response may include Retry-After, which tells the client how long to wait. A dispatcher that immediately retries 429 responses converts one constrained dependency into a retry storm. A dispatcher that sleeps inside every handler avoids the storm but strands worker capacity and makes shutdowns slow. Delayed redelivery is the cleaner boundary: record the attempt, release the worker, and make the message visible again no earlier than the allowed time.
One detail deserves skepticism. Retry-After can be expressed as seconds or as an HTTP date, and a response may omit it. Parse both forms; if it is absent or invalid, use capped exponential backoff with jitter. I'm not sure a universal cap exists because the right value depends on the support provider's published limits and the product SLO. Resolve that uncertainty with the provider contract and a load test, then put the chosen cap in configuration rather than scattering it through handlers.
Don't let each tenant consume the entire pool. If the queue holds password-reset follow-ups, ticket enrichment, and bulk transcript indexing, a noisy tenant's indexing batch can otherwise delay urgent replies. Per-tenant concurrency or fair partitions are more useful than a single large global worker count. Capacity planning should begin with the permitted request rate R, mean service time L, and a safety factor s: set effective concurrency near R * L * s, then validate it under bursty arrivals. The formula is a starting bound, not proof.
Keep it boring.
The DLQ is for messages that exhausted a bounded attempt policy or failed permanently; it must not become a second retry loop. AWS documents that a dead-letter queue receives messages after a configured receive threshold, and also warns that moving messages can affect ordering, which is a useful reminder even when a different implementation is used: replay is an operator action with business consequences. A malformed ticket reference belongs in quarantine with its job ID, tenant, attempt count, and final reason. A 429 belongs on a delayed path.
How should a SaaS app combine HTTP workers, retries, a DLQ, and delayed jobs?
Use one durable job envelope across the queue boundary. It needs a stable job ID, tenant ID, operation, creation time, attempt number, and payload reference. Avoid putting mutable credentials or an entire customer transcript in the message. The public webhook endpoint should authenticate the sender, reject an oversized body before decoding, and use the stable job ID as an idempotency key. A US job stays with the US worker and data plane; an EU job stays with the EU equivalents. Cron creates envelopes in each region but does no support work itself.
The safe processing order is receive, authenticate, claim the idempotency key, perform the business transition, persist the result, and acknowledge. The claim and business transition need a transaction or another atomic boundary. If the process stops after the transition but before acknowledgment, redelivery finds the completed key and returns success without repeating the customer-facing action.
Exactly-once delivery isn't the contract.
Effectively-once business effects are.
Here is a deliberately small Go handler. The storage interface hides the database transaction, while the response codes preserve the distinction between accepted work, retryable backpressure, and permanent input failure. The queue adapter should turn 429 into delayed redelivery, send repeated permanent failures to the DLQ according to policy, and acknowledge 204 or 200.
package worker
import (
"context"
"encoding/json"
"errors"
"net/http"
"strconv"
"time"
)
type Job struct {
ID string `json:"id"`
TenantID string `json:"tenant_id"`
Region string `json:"region"`
Operation string `json:"operation"`
Attempt int `json:"attempt"`
Payload json.RawMessage `json:"payload"`
}
var ErrRateLimited = errors.New("downstream rate limited")
type Store interface {
ApplyOnce(ctx context.Context, job Job) (alreadyApplied bool, err error)
}
type Handler struct {
Store Store
SharedSecret string
}
func (h Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodPost {
http.Error(w, "method not allowed", http.StatusMethodNotAllowed)
return
}
if r.Header.Get("Authorization") != "Bearer "+h.SharedSecret {
http.Error(w, "unauthorized", http.StatusUnauthorized)
return
}
r.Body = http.MaxBytesReader(w, r.Body, 256<<10)
var job Job
if err := json.NewDecoder(r.Body).Decode(&job); err != nil {
http.Error(w, "invalid job", http.StatusBadRequest)
return
}
if job.ID == "" || job.TenantID == "" || (job.Region != "US" && job.Region != "EU") {
http.Error(w, "invalid envelope", http.StatusUnprocessableEntity)
return
}
_, err := h.Store.ApplyOnce(r.Context(), job)
if errors.Is(err, ErrRateLimited) {
delay := retryDelay(job.Attempt)
w.Header().Set("Retry-After", strconv.Itoa(int(delay.Seconds())))
http.Error(w, "rate limited", http.StatusTooManyRequests)
return
}
if err != nil {
http.Error(w, "retry later", http.StatusServiceUnavailable)
return
}
w.WriteHeader(http.StatusNoContent)
}
func retryDelay(attempt int) time.Duration {
if attempt < 0 {
attempt = 0
}
if attempt > 6 {
attempt = 6
}
return time.Second * time.Duration(1<<attempt)
}
The example uses deterministic backoff to keep the mechanism visible. Production dispatchers should add bounded jitter so that a batch doesn't wake at one instant. Authentication also needs rotation and replay resistance appropriate to the threat model; a bearer secret illustrates the boundary but isn't a complete internet-facing webhook design. That's a limitation, not fine print.
Which queue ownership model fits the SLO?
The phrase best simple background job queue hides three separate decisions: who operates persistence, who controls dispatch, and where regional state lives. Put those into a buy-versus-build review before looking at feature matrices. Delayed delivery, bounded retries, a DLQ, cron-based enqueueing, and authenticated HTTP delivery are acceptance criteria. They are not enough on their own; the team also needs queue-age metrics, replay controls, regional isolation, and a documented restore path.
| Model | On-call load | Control and lock-in | Good fit | Reject when |
|---|---|---|---|---|
| Managed regional queue | Lowest broker burden; application paging remains | Service semantics and regional catalog constrain design | Small team, ordinary delayed jobs, HTTP worker adapter | Required region or delivery control is absent |
| Self-hosted broker | Team owns upgrades, storage, failover, and paging | Highest protocol and topology control | Existing broker expertise and a hard control requirement | No staffed broker rotation or tested restore |
| Database-backed dispatcher | Reuses familiar transactions and backup processes | Couples job pressure to application storage | Modest volume and strong need for atomic app writes | Queue scans or retention threaten the primary database SLO |
A managed service isn't automatically simpler if every job needs a custom bridge to reach a public webhook endpoint. Conversely, a database table isn't automatically cheap when polling, vacuum pressure, and retention compete with interactive customer-support traffic. Include engineering time and on-call interruption in the cost model, but don't pretend those estimates are precise. Your mileage may vary with existing expertise.
Stick with the database-backed model when throughput is modest, the team already has a transactional outbox, and coupling is acceptable. Choose self-hosting when broker behavior or deployment control is a hard requirement and there is a staffed rotation. Otherwise, managed regional persistence plus a thin HTTP dispatcher generally minimizes the platform surface. This is a decision rule, not a vendor ranking.
How do you verify draining, retries, and rollback?
Test the failure policy before raising concurrency. Seed a staging queue with duplicate job IDs, one permanently invalid envelope, and a burst that causes controlled 429 responses. Verify that duplicate IDs produce one business transition, invalid work reaches quarantine only after the declared policy, Retry-After delays the next delivery, and queue age returns toward the SLO after the limit is removed. Also confirm that US and EU jobs never cross their assigned worker pools. Counts alone won't prove this; trace a stable job ID through enqueue, delivery, persistence, acknowledgment, and replay.
The minimum dashboard needs oldest-ready-job age, ready depth, in-flight count, attempt distribution, 429 rate, permanent rejection rate, DLQ depth, and completed jobs by region and tenant. Page on user-visible risk, especially queue age against the support SLO, rather than raw depth. A backlog of ten long-running transcript jobs may be worse than a thousand tiny enrichments, so depth without age and service time is weak evidence.
Deploy the worker with concurrency set below the calculated bound. Increase it in steps while watching downstream 429s and oldest-job age. If 429s rise without a corresponding drain-rate gain, step back; more workers are only producing retries. Rollback means pausing new dispatch, allowing in-flight requests a bounded shutdown window, deploying the prior worker version, and resuming at the last known-safe concurrency.
Don't purge.
Preserve original envelopes and idempotency records so redelivery remains harmless.
Run a DLQ replay drill with a small reviewed batch, not the entire queue. Confirm the underlying cause is no longer present, preserve the original job ID, set a replay annotation, and rate-limit the replay separately from live support work. Stop if queue age or 429 rate breaches the rollout threshold. A replay button without a stop condition is just another incident trigger.
One last gate: test the cron trigger twice for the same logical window. If it creates two envelopes, they must share a business idempotency key or collapse before the customer-facing transition. Schedulers overlap during deploys and recoveries; pretending they never do leaves correctness to timing.
References
- MDN, HTTP 429 Too Many Requests: https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/429
Further reading
- AWS, Using dead-letter queues: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html
Top comments (0)