DEV Community

Renato Silva
Renato Silva

Posted on

MQTT QoS Levels Will Lie to You at 50k Devices

🔧 The Problem

Somewhere around 15,000 concurrently connected devices, we stopped trusting QoS numbers. Not because the spec is wrong — MQTT's QoS guarantees are precisely defined and the broker we ran (EMQX, later supplemented with a VerneMQ cluster for comparison) implemented them correctly. The problem is that "QoS 1 means at-least-once delivery" is a sentence that only means something nice when you ignore reconnect storms, session expiry, and the fact that your clients are cheap microcontrollers with flaky cellular modems.

At small scale, QoS 1 and QoS 2 behave exactly like the textbook says. At 50k devices, with real-world network jitter, the guarantees still technically hold — but the conditions under which they hold become so narrow that in practice you get duplicate storms, silent message loss during session takeover, and retry amplification that looks indistinguishable from a DDoS against your own broker.

This post is about what actually broke, what we changed in broker config, and which metrics told us trouble was coming before clients started timing out.

📉 QoS 1: "At Least Once" Becomes "At Least However Many Times The Network Felt Like"

QoS 1 works like this: publisher sends PUBLISH, waits for PUBACK, retransmits if it doesn't get one in time. Simple. The lie is in "in time."

Our fleet was mostly battery-powered sensors on LTE-M and NB-IoT, in regions with inconsistent coverage. A PUBACK getting delayed by 4-6 seconds during a cell handover was routine, not exceptional. Client firmware (reasonably) had a retry timeout of 3 seconds. Result: the broker would receive the same QoS 1 publish 2-3 times for a single sensor reading, because the ack was in flight but the client gave up waiting.

This isn't a spec violation — QoS 1 explicitly allows duplicates. But downstream consumers that weren't idempotent choked on it. We had a time-series pipeline that happily double/triple-counted motion events during network-degraded periods, which is exactly when you most need the data to be right.

Fix was two-pronged:

  1. Dedup at the ingestion layer using (client_id, packet_id, timestamp-bucket), not relying on broker-side guarantees at all.
  2. Tune the retry/ack window per device class. Our default was a global constant, which is fine for devices on stable wifi and actively harmful for cellular devices in weak coverage.

erlang
%% emqx.conf snippet - per-listener ack timeout tuning
listener.tcp.cellular {
bind = "0.0.0.0:1884"
max_connections = 20000
mqtt.retry_interval = "8s"
mqtt.max_inflight = 8
mqtt.max_awaiting_rel = 32
}

listener.tcp.wifi {
bind = "0.0.0.0:1883"
max_connections = 40000
mqtt.retry_interval = "3s"
mqtt.max_inflight = 32
}

Splitting listeners by device class so we could tune retry/inflight windows independently cut duplicate publishes by roughly 60% without touching firmware.

🔁 QoS 2: Exactly Once, Except When Sessions Expire

QoS 2 is the four-way handshake (PUBLISH → PUBREC → PUBREL → PUBCOMP) that's supposed to guarantee exactly-once delivery. It does, as long as the session state spanning that handshake survives. The lie is that "the session" is a much more fragile thing than people assume under reconnect storms.

Here's the sequence that bit us: device drops off network mid-handshake (common during a cell tower failover affecting thousands of devices simultaneously — a literal reconnect storm). It reconnects with clean_session=false expecting the broker to resume the QoS 2 flow. But our session expiry interval was tuned for normal operation, not for a correlated outage affecting 8,000 devices at once. By the time a chunk of them reconnected, their sessions had expired and been garbage collected to protect broker memory. The in-flight QoS 2 message was just gone. No error, no DISCONNECT reason code that pointed at it directly — just a message that should have been guaranteed and wasn't.

Exactly-once only holds if the session actually outlives the gap. During a reconnect storm, the gap is exactly what you can't predict.

bash

vernemq.conf - what we actually changed

persistent_client_expiration = 4h
max_inflight_messages = 20
max_online_messages = 5000
allow_register_during_netsplit = off

We bumped persistent_client_expiration way up, which cost us memory (session state for a disconnected client isn't free), so we paired it with a hard cap on max_online_messages to stop a single misbehaving device from hoarding broker RAM while "temporarily" offline for six hours.

The other change was more philosophical than technical: we stopped using QoS 2 for anything except firmware update acknowledgments. For telemetry, QoS 1 + idempotent ingestion gives you the same practical outcome with a fraction of the broker-side bookkeeping and none of the four-way handshake overhead at scale. QoS 2's exactly-once guarantee is real, but the cost of maintaining it under load (per-message state tracking, max_awaiting_rel limits, handshake retransmission) is a tax you should only pay where duplicates are genuinely unacceptable.

📊 The Metrics That Actually Predicted Trouble

Connection count and message throughput are vanity metrics for this failure mode. They stay flat right up until things fall over. What actually gave us lead time:

  • mqtt.packet.puback.missed rate per listener (or the equivalent dropped/duplicate ack counter in your broker) — a creeping rise here, even while everything else looks fine, is the first sign of network conditions degrading faster than your retry tuning accounts for.
  • Inflight queue depth percentiles, not just connection count. p99 inflight queue depth climbing while p50 stays flat means a subset of devices are in a degraded-network pocket, which is exactly the population that will trigger a correlated reconnect storm later.
  • Session takeover rate — how often a device reconnects to find a session already registered under its client ID (stale TCP connection, duplicate device, or clock skew causing premature reconnect). This spikes hard right before cascading disconnects.
  • Broker-side memory attributable to persistent sessions, tracked separately from message queue memory. This is the metric that tells you whether your persistent_client_expiration tuning is actually sustainable or just deferring an OOM.

promql

alerting rule we actually use

rate(emqx_messages_qos2_dropped_total[5m]) > 0.5
and
histogram_quantile(0.99, emqx_mqtt_inflight_depth_bucket) > 25

That combined alert — rising QoS2 drops and p99 inflight depth — gives us roughly 10-15 minutes of lead time before a reconnect storm turns into a cascading outage. Either signal alone is noisy; together they're a reliable precursor.

🤔 Where This Leaves Us

QoS levels are correct specifications for a single publisher-subscriber pair in isolation. They were never a promise about fleet behavior under correlated failure, and treating them as one is the actual root cause of most of the pain described here. The fix wasn't "use a better QoS level" — it was accepting that reliability at fleet scale is a property you build at the ingestion and session-management layer, with QoS as one input rather than the whole contract.

If you're running MQTT at scale: what's your broker's session expiry tuned to, and have you actually tested it against a simulated reconnect storm rather than just steady-state load? That gap between "load tested" and "failure tested" is where most of these lies hide.

Top comments (0)