Short answer: put every outbound API call behind a durable queue, retry only errors that can recover, and move poison messages to a dead-letter queue (DLQ) with a measured redrive path. For a B2B SaaS worker pool, this keeps rate limits from turning into a runaway backlog while preserving a clear latency-versus-cost choice.
The useful unit is a job, not a request.
A scheduler records the customer, endpoint, idempotency key, and next-attempt time. Workers claim jobs with a lease, call the API, and either acknowledge success or write a retry decision. A separate DLQ stores messages that exhausted policy or failed validation. Monitoring then connects queue age, retry volume, and provider responses instead of treating a single 429 as an outage. That ledger is the foundation for a defensible redrive decision: you can explain which tenant was delayed, why it was delayed, and what work will be released next.
Start with a failure ledger, not a bigger worker pool
Start with an explicit error taxonomy. A 429 with a Retry-After header is usually delayable; honor the header, cap the delay, and add jitter so a whole pool does not wake together. A network timeout can be retried when the operation is idempotent. A 400 caused by a malformed payload should go straight to the DLQ. A 401 needs credential rotation or an operator decision, not an infinite loop.
Here is a small TypeScript policy. It is deliberately boring: the queue adapter can be SQS, Redis, Postgres, or a self-hosted broker, as long as it offers visibility timeouts or leases and an atomic acknowledgement.
type Failure = { status?: number; retryAfterMs?: number; timeout?: boolean };
function nextDelay(attempt: number, failure: Failure): number | null {
if (failure.status === 429) {
const serverDelay = failure.retryAfterMs ?? 1000;
return Math.min(serverDelay + Math.random() * 250, 60_000);
}
if (failure.timeout && attempt < 5) {
const exponential = Math.min(500 * 2 ** attempt, 30_000);
return exponential + Math.random() * 200;
}
if (failure.status !== undefined && failure.status >= 500 && attempt < 4) {
return Math.min(1000 * 2 ** attempt, 60_000);
}
return null;
}
The worker increments an attempt counter in the same transaction that schedules the next visibility time. When nextDelay returns null, it writes a reason code and the original payload reference to the DLQ, then acknowledges the source message. That ordering matters: acknowledging first can lose a job; publishing first without an idempotency key can duplicate it.
How can queue retry and DLQ redrive keep rate-limited calls bounded?
Redrive is a controlled release, not a button that empties a bucket. Store the failure class, first-seen timestamp, and a hash of the request. An operator can select one class, cap messages per minute, and send them back with a fresh attempt budget. Keep the DLQ immutable for audit, and write a redrive event for every message.
Backlog math gives a practical guardrail. If 2,400 jobs are waiting and the pool safely completes 20 jobs per second, the ideal drain time is 120 seconds; rate-limit pauses, retries, and new arrivals make the real value larger. Alert on age of the oldest ready job, not only depth. Depth can look stable while old tenants starve behind newly arriving work.
Measure twice.
I once treated a flat backlog graph as good news. It was not. New work was arriving at almost the same rate as completions, while the oldest job kept aging. The fix was a per-tenant age metric and a low-watermark redrive, not more workers. That change also exposed a less obvious cost: retries were consuming the same provider quota as fresh work, so the apparent throughput was masking a growing token bill. We split the dashboard into admission, completion, and retry lanes, then set an alert on retry amplification. The queue finally told us whether it was slow because the provider was closed for the window or because our payloads were bad.
Use a token bucket or leaky bucket around the provider limit. Keep concurrency and token rate separate: concurrency controls in-flight latency, while tokens control admission. If the provider exposes a daily quota, reserve tokens for interactive jobs and let batch work absorb the delay. Your mileage may vary when the provider uses undocumented sliding windows; measure response headers and keep the limiter configurable.
| Signal | Meaning | Action |
|---|---|---|
| Oldest ready age | User-visible staleness | Slow intake or add capacity |
| 429 rate | Admission exceeds provider window | Lower token rate; honor Retry-After
|
| DLQ growth by reason | Poison data or policy mismatch | Fix producer or redrive a narrow class |
| Retry amplification | Cost and latency multiplier | Tighten attempts or backoff |
Redrive review: which safeguards belong in production?
Make handlers idempotent. Include an idempotency key derived from tenant, logical operation, and source version; never derive it from a random retry id. A lease must expire, and a worker must renew it for long API calls. On process shutdown, stop claiming new work, finish bounded calls, and let the lease return anything unfinished.
Keep secrets out of messages and logs. Workers should fetch credentials through a secret manager, and logs should record a key identifier rather than the key itself. OWASP's key-management guidance is a useful baseline for rotation and access separation.
Testing needs time control. Inject a clock and a fake provider that emits 429, timeout, 400, and 500 sequences. Assert that a 400 reaches the DLQ once, that a 429 delay never exceeds the cap, and that redrive preserves the idempotency key. Run a small load test with a deliberately low quota; a fast local test can hide synchronization bursts.
The catch is that this design is not suitable when jobs cannot be made idempotent or when strict ordering spans tenants. Use a workflow engine with explicit compensation for those cases, or keep a single ordered partition and accept higher latency. Also stick with a simple cron plus a database when volume is tiny; operating a broker and DLQ adds real maintenance cost.
Ship the smallest observable loop first: enqueue, lease, call, classify, retry, DLQ, and redrive. Add dashboards for oldest age and reason-coded failures before increasing concurrency. That sequence exposes whether the bottleneck is provider quota, payload quality, or your own worker pool.
Top comments (0)