A 90-second upstream blip doesn't take you down. What your retries do in the ninety seconds after it ends does.
We learned this the expensive way. A vendor API returned 502s for about a minute and a half. Three thousand executions were queued behind it across 40 workers. When the vendor recovered, every worker fired at the same cadence, one second apart, forever synchronized, because our retry loop was sleep(1) with no jitter. p99 went 200ms → 18s. The vendor's rate limiter banned our egress IP for ten minutes. Our Postgres pool of 20 connections was gone, held by executions parked on socket timeouts.
Three separate defects, none of them in the vendor's code:
- Correlated retries. Fixed-interval backoff keeps a herd in lockstep. It never decorrelates.
- No shared failure state. Each worker independently concluded the upstream was healthy.
- No cap on aggregate in-flight work. Retries amplified load exactly when the downstream was weakest.
What n8n gives you, and where it stops
n8n's node-level Retry On Fail (Max Tries / Wait Between Tries) is a fixed delay, applied per item, inside one worker process. In queue mode (EXECUTIONS_MODE=queue) that means retry state is per-process and per-execution: no jitter, no cross-worker breaker, and no way to distinguish "this item failed" from "this dependency is down." Continue On Fail keeps the workflow moving; it does not stop the fan-out. The fix has to live in the HTTP client your Code node calls, not in node settings.
The coarse safety valve is worker concurrency: n8n worker --concurrency=10 × 40 workers = 400 in-flight ceiling. That number has to be one the downstream will tolerate, not one your CPU can handle.
Deliverable 1 — full-jitter backoff plus a breaker that actually opens
import random, time, httpx
RETRYABLE = {429, 500, 502, 503, 504}
class Breaker:
def __init__(self, threshold=5, reset_after=30.0):
self.threshold, self.reset_after = threshold, reset_after
self.fails, self.opened_at = 0, 0.0
def allow(self):
if self.fails < self.threshold:
return True
if time.monotonic() - self.opened_at >= self.reset_after:
self.fails = self.threshold - 1 # half-open: one probe through
return True
return False
def record(self, ok):
if ok:
self.fails = 0
return
self.fails += 1
if self.fails >= self.threshold:
self.opened_at = time.monotonic()
def request(client, method, url, *, attempts=5, base=0.4, cap=20.0, breaker=None, **kw):
err = None
for i in range(attempts):
if breaker and not breaker.allow():
raise RuntimeError(f"circuit open: {url}")
try:
r = client.request(method, url, **kw)
except (httpx.ConnectError, httpx.ConnectTimeout, httpx.ReadTimeout) as exc:
if breaker: breaker.record(False)
err = exc
else:
if breaker: breaker.record(r.status_code < 500)
if r.status_code not in RETRYABLE:
return r
err = r.status_code
if r.status_code == 429 and (ra := r.headers.get("Retry-After")):
time.sleep(min(float(ra), cap))
continue
if i == attempts - 1:
break
time.sleep(random.uniform(0, min(cap, base * 2 ** i))) # full jitter
raise RuntimeError(f"exhausted {attempts} attempts: {err}")
Full jitter (random(0, min(cap, base·2^i))) flattens the herd into a smooth arrival curve instead of the multi-modal spike decorrelated jitter produces. Retry-After wins over your schedule when present. And never retry a POST without an idempotency key at the transport layer — that's how you get double charges, not resilience.
Deliverable 2 — exact sliding window in one atomic Lua eval
Check-and-increment must be atomic. INCR + EXPIRE is two round trips and leaks counters on crash; fixed-window counters permit a 2× burst across the boundary. A sorted set is exact:
-- KEYS[1] = window key ARGV = now_ms, window_ms, limit, member
local now = tonumber(ARGV[1])
local window = tonumber(ARGV[2])
local limit = tonumber(ARGV[3])
redis.call('ZREMRANGEBYSCORE', KEYS[1], 0, now - window)
if redis.call('ZCARD', KEYS[1]) >= limit then
local oldest = redis.call('ZRANGE', KEYS[1], 0, 0, 'WITHSCORES')
return {0, tonumber(oldest[2]) + window - now}
end
redis.call('ZADD', KEYS[1], now, ARGV[4])
redis.call('PEXPIRE', KEYS[1], window)
return {1, 0}
import time, uuid, redis.asyncio as redis
from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse
app = FastAPI()
r = redis.Redis(host="127.0.0.1", port=6379, decode_responses=True)
window = r.register_script(open("sliding_window.lua").read())
@app.middleware("http")
async def rate_limit(request: Request, call_next):
tenant = request.headers.get("x-tenant", "anon")
key = f"{{rl:{tenant}}}:{request.url.path}" # hash tag = cluster-safe
allowed, retry_ms = await window(
keys=[key],
args=[int(time.time() * 1000), 60_000, 120, uuid.uuid4().hex],
)
if not allowed:
return JSONResponse(
{"error": "rate_limited"}, status_code=429,
headers={"Retry-After": str(max(1, int(retry_ms / 1000)))},
)
return await call_next(request)
One member per request inside the window is the storage cost, bounded by the limit itself. The {rl:tenant} hash tag keeps a tenant's keys in one Redis Cluster slot — drop it and EVAL fails cross-slot under Cluster.
Wiring it into a workflow
A Code node cannot hold breaker state across items, and in queue mode there are N worker processes anyway — in-memory state is a lie. Put the breaker in Redis and call the wrapper from a sidecar service your workflow hits, rather than reimplementing the logic per workflow. Then set Retry On Fail to Max Tries = 1 on the node itself. Node retry × client retry multiplies: 3 × 5 attempts is fifteen calls per item, which is the amplification that started this.
After the change: retries spread across 0–20s, peak arrival rate down roughly 8×, no ban, 60% pool headroom at steady state. The tradeoff you inherit is that an open breaker fails fast and silent — surface it as its own alert state, distinct from generic node errors, or you'll drop work while the dashboard stays green.
Top comments (1)
The third defect is the one most retry guides skip — no cap on aggregate in-flight work. Jitter decorrelates the herd, but if the downstream is still down, decorrelated retries are just a slower stampede. The piece I'd add from building email-sending pipelines: make the retried operation safe to repeat, not just polite about timing. Fixed-interval retries plus a non-idempotent send path is how you turn a 90-second blip into duplicate deliveries. I treat "is this safe to retry" as a property of the operation, decided before the first send, not the fifth. Do you gate retries on idempotency at the HTTP-client layer, or is that still the operator's problem?