When a broker goes down and a thousand edge devices all try to reconnect every second, the moment the broker comes back online it gets hit with a thousand simultaneous connection requests and dies again. This is the thundering herd problem, and it is entirely self-inflicted. The fix is exponential backoff with jitter — a reconnection strategy every networked embedded application should implement from day one.
🔗 See project on GitHub: Resilient Edge MQTT Client
The Wrong Way
# Every device retries every second. 1000 devices = 1000 req/s on reconnect.
while True:
try:
client.connect(broker_host, broker_port)
break
except Exception:
time.sleep(1) # Fixed delay — all devices retry in lockstep
Fixed-delay retry is the problem. Every device that disconnected at the same time will retry at exactly the same time, forever, until one of them happens to succeed. At scale, this makes outages self-perpetuating.
The Right Way
import time
import random
def connect_with_backoff(client, broker_host, broker_port,
min_delay=1, max_delay=300):
"""
Reconnect with exponential backoff and jitter.
Delay sequence (approximate): 1s, 2s, 4s, 8s, 16s, 32s, 60s, 60s...
Jitter adds ±10% randomness so devices that disconnected together
do not retry together.
"""
delay = min_delay
while True:
try:
# Jitter is proportional to the current delay, not fixed.
# At delay=1 it adds up to 0.1s. At delay=60 it adds up to 6s.
# This keeps devices spread out even after long outages.
jitter = random.uniform(0, delay * 0.1)
time.sleep(delay + jitter)
client.connect(broker_host, broker_port)
return # Success — caller resets delay to min_delay
except Exception as e:
print(f"Connection failed: {e}. Retrying in {delay:.1f}s")
delay = min(delay * 2, max_delay)
The three decisions baked into this function are worth understanding individually.
Doubling the delay (the exponential part) means the load on the broker drops by half with each retry cycle. After a few rounds, devices are spread so far apart in time that the broker recovers comfortably before the next wave arrives.
Capping at max_delay prevents devices from backing off indefinitely. A cap of 60–300 seconds is usually right for embedded systems — long enough to give the broker real recovery time, short enough that you do not lose hours of data during a brief outage.
Proportional jitter is the part most implementations get wrong. Adding a small fixed jitter like random.uniform(0, 0.5) works when delay is 1 second but is meaningless when delay is 60 seconds — all devices are still retrying within the same half-second window. Making jitter proportional to the current delay (10% here) keeps devices spread out at every stage of backoff.
Three Ways to Add Jitter
The 10% proportional jitter above is deliberately conservative — it keeps retries close to the intended backoff curve while still desynchronizing devices. But it's not the only approach, and for larger fleets more aggressive strategies spread load better. These are the three established patterns, all built on the same base calculation:
temp = min(cap, base * (2 ** attempt)) # uncapped exponential value
Full Jitter picks the delay uniformly between zero and the full exponential value. This maximizes spread but sacrifices any guaranteed minimum wait — some devices will retry almost immediately.
sleep = random.uniform(0, temp)
Equal Jitter keeps half the delay fixed and randomizes the other half. This guarantees a minimum wait while still breaking synchronization — a good middle ground when you don't want devices hammering back instantly.
sleep = temp / 2 + random.uniform(0, temp / 2)
Decorrelated Jitter bases each delay on the previous sleep time rather than the attempt count, letting the backoff wander upward more smoothly. AWS's analysis found this among the most effective at minimizing total work against a recovering service.
sleep = min(cap, random.uniform(base, prev_sleep * 3))
For a small edge fleet, proportional or equal jitter is plenty. For thousands of clients hitting a shared API, full or decorrelated jitter spreads the recovery load more evenly.
Reset the Delay on Success
This is the step that gets forgotten. After a successful connection, the delay must reset to min_delay. If it stays at whatever value it reached during the outage, the next disconnection — even a brief one — will start with a 60-second wait instead of a 1-second wait.
class ResilientMQTTClient:
def __init__(self, broker_host, broker_port):
self.broker_host = broker_host
self.broker_port = broker_port
self.min_delay = 1
self.max_delay = 300
self.current_delay = self.min_delay # Reset this on every successful connect
def on_connect(self, client, userdata, flags, rc):
if rc == 0:
self.current_delay = self.min_delay # Always reset here
print("Connected — backoff delay reset to minimum")
def reconnect(self):
jitter = random.uniform(0, self.current_delay * 0.1)
time.sleep(self.current_delay + jitter)
try:
self.client.reconnect()
except Exception:
self.current_delay = min(self.current_delay * 2, self.max_delay)
With Paho MQTT, the right place to reset is inside the on_connect callback, which fires on the background network thread whenever a connection is established. Resetting in the reconnection loop itself is too early — the connection might still fail after the reset.
Add a Restart Rate Limit at the System Level
Backoff handles network-level reconnection. But if your application itself is crashing and restarting — not just reconnecting — you need a second line of defence. systemd's StartLimitBurst and StartLimitIntervalSec directives cap how many times a service can restart within a given window before systemd stops trying and marks it failed.
[Service]
Restart=on-failure
RestartSec=5 # Base delay before first restart
StartLimitIntervalSec=120 # Window for counting restarts
StartLimitBurst=5 # Give up after 5 restarts in 120 seconds
This prevents a crashing service from hammering a broker with rapid reconnections that bypass your application-level backoff entirely.
Don't Retry Blindly
Backoff controls when you retry. It says nothing about whether you should — and retrying the wrong things causes worse problems than the outage itself.
Not every error is retryable. Retry only on transient failures: connection timeouts and 5xx responses like 500 and 503, which mean the service is temporarily unwell and may recover. Do not retry client errors like 401 or 404 — a request that's unauthorized or points at something that doesn't exist will fail identically no matter how many times you send it. Retrying it just wastes cycles and pollutes your logs.
Non-idempotent operations need protection. If a retry can duplicate a side effect — charging a card, creating a record, sending a message — then a retry that succeeds after the original silently went through will corrupt your data. The standard fix is an idempotency key: the client attaches a unique token to the request, and the server uses it to recognize and deduplicate a retry of an operation it already completed. Reconnecting to a broker is naturally idempotent; posting a payment is not.
Always set an attempt limit. The max_delay cap stops any single wait from growing unbounded, but a client can still retry a capped delay forever. For request/response operations, give up after a fixed number of attempts (typically 5–7) and surface the error to the caller rather than retrying into the void. Persistent connections like MQTT are the exception — there, retrying indefinitely against max_delay is usually what you want.
Quick Reference
Enable exponential backoff on every service that connects to an external broker, API, or database — not just MQTT. Start with min_delay=1, max_delay=60 for most edge applications. Use proportional jitter (10% of current delay) for small fleets, or full/decorrelated jitter when thousands of clients share a service — never fixed jitter. Always reset the delay counter inside the successful-connection callback. Retry only transient errors (timeouts, 5xx), never client errors (4xx). Protect non-idempotent operations with an idempotency key, and cap request/response retries at 5–7 attempts. Pair application-level backoff with systemd restart rate limiting for complete coverage.
If you found this useful, drop a ❤️ or a comment — I'd love to hear how you handle reconnection in your own systems.
Top comments (0)