A 90-second upstream blip doesn't take you down. What your retries do in the ninety seconds after it ends does.
We learned this the expensive way. A vendor API returned 502s for about a minute and a half. Three thousand executions were queued behind it across 40 workers. When the vendor recovered, every worker fired at the same cadence, one second apart, forever synchronized, because our retry loop was sleep(1) with no jitter. p99 went 200ms → 18s. The vendor's rate limiter banned our egress IP for ten minutes. Our Postgres pool of 20 connections was gone, held by executions parked on socket timeouts.
Three separate defects, none of them in the vendor's code:
- Correlated retries. Fixed-interval backoff keeps a herd in lockstep. It never decorrelates.
- No shared failure state. Each worker independently concluded the upstream was healthy.
- No cap on aggregate in-flight work. Retries amplified load exactly when the downstream was weakest.
What n8n gives you, and where it stops
n8n's node-level Retry On Fail (Max Tries / Wait Between Tries) is a fixed delay, applied per item, inside one worker process. In queue mode (EXECUTIONS_MODE=queue) that means retry state is per-process and per-execution: no jitter, no cross-worker breaker, and no way to distinguish "this item failed" from "this dependency is down." Continue On Fail keeps the workflow moving; it does not stop the fan-out. The fix has to live in the HTTP client your Code node calls, not in node settings.
The coarse safety valve is worker concurrency: n8n worker --concurrency=10 × 40 workers = 400 in-flight ceiling. That number has to be one the downstream will tolerate, not one your CPU can handle.
Deliverable 1 — full-jitter backoff plus a breaker that actually opens
import random, time, httpx
RETRYABLE = {429, 500, 502, 503, 504}
class Breaker:
def __init__(self, threshold=5, reset_after=30.0):
self.threshold, self.reset_after = threshold, reset_after
self.fails, self.opened_at = 0, 0.0
def allow(self):
if self.fails < self.threshold:
return True
if time.monotonic() - self.opened_at >= self.reset_after:
self.fails = self.threshold - 1 # half-open: one probe through
return True
return False
def record(self, ok):
if ok:
self.fails = 0
return
self.fails += 1
if self.fails >= self.threshold:
self.opened_at = time.monotonic()
def request(client, method, url, *, attempts=5, base=0.4, cap=20.0, breaker=None, **kw):
err = None
for i in range(attempts):
if breaker and not breaker.allow():
raise RuntimeError(f"circuit open: {url}")
try:
r = client.request(method, url, **kw)
except (httpx.ConnectError, httpx.ConnectTimeout, httpx.ReadTimeout) as exc:
if breaker: breaker.record(False)
err = exc
else:
if breaker: breaker.record(r.status_code < 500)
if r.status_code not in RETRYABLE:
return r
err = r.status_code
if r.status_code == 429 and (ra := r.headers.get("Retry-After")):
time.sleep(min(float(ra), cap))
continue
if i == attempts - 1:
break
time.sleep(random.uniform(0, min(cap, base * 2 ** i))) # full jitter
raise RuntimeError(f"exhausted {attempts} attempts: {err}")
Full jitter (random(0, min(cap, base·2^i))) flattens the herd into a smooth arrival curve instead of the multi-modal spike decorrelated jitter produces. Retry-After wins over your schedule when present. And never retry a POST without an idempotency key at the transport layer — that's how you get double charges, not resilience.
Deliverable 2 — exact sliding window in one atomic Lua eval
Check-and-increment must be atomic. INCR + EXPIRE is two round trips and leaks counters on crash; fixed-window counters permit a 2× burst across the boundary. A sorted set is exact:
-- KEYS[1] = window key ARGV = now_ms, window_ms, limit, member
local now = tonumber(ARGV[1])
local window = tonumber(ARGV[2])
local limit = tonumber(ARGV[3])
redis.call('ZREMRANGEBYSCORE', KEYS[1], 0, now - window)
if redis.call('ZCARD', KEYS[1]) >= limit then
local oldest = redis.call('ZRANGE', KEYS[1], 0, 0, 'WITHSCORES')
return {0, tonumber(oldest[2]) + window - now}
end
redis.call('ZADD', KEYS[1], now, ARGV[4])
redis.call('PEXPIRE', KEYS[1], window)
return {1, 0}
import time, uuid, redis.asyncio as redis
from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse
app = FastAPI()
r = redis.Redis(host="127.0.0.1", port=6379, decode_responses=True)
window = r.register_script(open("sliding_window.lua").read())
@app.middleware("http")
async def rate_limit(request: Request, call_next):
tenant = request.headers.get("x-tenant", "anon")
key = f"{{rl:{tenant}}}:{request.url.path}" # hash tag = cluster-safe
allowed, retry_ms = await window(
keys=[key],
args=[int(time.time() * 1000), 60_000, 120, uuid.uuid4().hex],
)
if not allowed:
return JSONResponse(
{"error": "rate_limited"}, status_code=429,
headers={"Retry-After": str(max(1, int(retry_ms / 1000)))},
)
return await call_next(request)
One member per request inside the window is the storage cost, bounded by the limit itself. The {rl:tenant} hash tag keeps a tenant's keys in one Redis Cluster slot — drop it and EVAL fails cross-slot under Cluster.
Wiring it into a workflow
A Code node cannot hold breaker state across items, and in queue mode there are N worker processes anyway — in-memory state is a lie. Put the breaker in Redis and call the wrapper from a sidecar service your workflow hits, rather than reimplementing the logic per workflow. Then set Retry On Fail to Max Tries = 1 on the node itself. Node retry × client retry multiplies: 3 × 5 attempts is fifteen calls per item, which is the amplification that started this.
After the change: retries spread across 0–20s, peak arrival rate down roughly 8×, no ban, 60% pool headroom at steady state. The tradeoff you inherit is that an open breaker fails fast and silent — surface it as its own alert state, distinct from generic node errors, or you'll drop work while the dashboard stays green.
Top comments (3)
The third defect is the one most retry guides skip — no cap on aggregate in-flight work. Jitter decorrelates the herd, but if the downstream is still down, decorrelated retries are just a slower stampede. The piece I'd add from building email-sending pipelines: make the retried operation safe to repeat, not just polite about timing. Fixed-interval retries plus a non-idempotent send path is how you turn a 90-second blip into duplicate deliveries. I treat "is this safe to retry" as a property of the operation, decided before the first send, not the fifth. Do you gate retries on idempotency at the HTTP-client layer, or is that still the operator's problem?
That distinction between "polite timing" and "safe execution" is the crux of it. Leaving idempotency to the operator or downstream workflow author almost always leads to duplicate payments or double-sent emails the first time an upstream times out after processing.
In our stack, we gate retries strictly at the transport/gateway interceptor layer:
GET/HEAD/PUT/DELETEare safe to retry with backoff.POSTrequests are strictly non-retryable by default unless the request carries an explicitIdempotency-Keyheader (derived deterministically from the payload hash or task execution ID).If a downstream throws a 502/504 on a non-idempotent
POST, the pipeline immediately trips to a Dead-Letter Queue (DLQ) rather than allowing automated loops. Swallowing an error into a retry loop without a deduplication key is just automating disaster.The POST-non-retryable-unless-keyed rule is exactly where I landed building email-sending pipelines — same failure shape, worse stakes. The timeout-after-processing case is the one that burns everyone once: the upstream processed the send, the client only saw the timeout, and the retry double-delivers. I now key every send at ingest (deterministic hash of payload plus recipient plus window) and let the DLQ carry the ambiguity, because a queued duplicate is diagnosable and a sent duplicate is a support ticket. The part I still wrestle with: key collision when the same logical operation legitimately repeats — how long do you hold your idempotency window open before the store cost outweighs the protection?****