MQTT load testing: thousands of devices, reconnect storms and what QoS really measures

What stresses an MQTT broker is often not the message rate but the number of connections. Ten thousand devices, each sending a small message every few seconds, means ten thousand sessions, subscriptions and keep-alives the broker holds in memory. A broker that handles 5,000 messages a second from 20 connections without blinking can fall over at a tenth of that rate from 10,000 devices.

An MQTT load test answers three questions:

1. How many concurrent devices does the broker accept?
2. How long does a message's acknowledgement take at that load?
3. What happens when every device reconnects at once after an outage?

Disclosure: I build Spitfire, a self-hosted load testing tool. The method below applies to any tool; the last section shows how Spitfire does it.

What to measure

• Concurrent connections: clients connected at once. Each one costs memory, a file descriptor and keep-alive traffic; the limit often comes here before CPU.
• Connect rate: new connections accepted per second. The TLS handshake and authentication are expensive. Thousands of devices reconnecting together after a network outage (a reconnect storm) is a bottleneck that a steady load never shows.
• Publish latency: it depends on QoS, so use the QoS your devices really use.
• Fan-out: a message is copied to every matching subscriber. A few services subscribed to a broad filter such as devices/+/telemetry multiply what the broker sends.
• Errors: refused connections, authentication failures, timeouts and sessions the broker closes.

What QoS really measures

"Publish latency" means something different at each QoS level:

• QoS 0: the client writes the message to the socket and moves on. There is no answer from the broker, so the number you get is how fast your own client writes, not how fast the broker accepts.
• QoS 1: the time to the broker's PUBACK. This is the one most IoT fleets care about.
• QoS 2: the time to PUBCOMP, at the end of a four-step handshake. Expect it to be clearly slower than QoS 1, and expect the broker to hold more state per in-flight message.

Testing at QoS 0 because "it is faster" and then running production at QoS 1 gives you a number that describes neither.

Shaping the load

The natural model is one virtual user per device: each VU opens its own connection, sends a message at intervals and stays connected. The message rate follows from the device count and the interval: 500 devices sending every 5 seconds is 100 messages a second.

Give the interval a little randomness (say 4–6 s) instead of a fixed value. Otherwise every device sends at the same moment and you get waves that do not happen in reality.

The ramp-up sets the connect rate. Reaching 500 devices in 2 minutes is about 4 connections a second; reaching the same number in 10 seconds is a reconnect storm. Run them as separate tests: one shows the broker under steady load, the other how it recovers after an outage. To find the device count where the broker breaks, raise it in steps.

Common mistakes

• The same client id. A second client connecting with the same client id drops the first one's session. Give every device its own id, and if two tests hit the same broker at once, give their ids different prefixes. Otherwise you are testing session takeover, endlessly.
• Many messages from few connections. 10,000 messages a second over 10 connections stresses the broker very differently from ten thousand devices. Keep the device count close to reality and set the rate with the interval.
• Retained messages and persistent sessions. Messages sent with retain, and sessions without clean session, stay on the broker after the test. Use a separate topic tree for the test and plan the cleanup.
• The load generator's own limit. Every connection is a socket on the generating machine too. For thousands of devices raise the file descriptor limit and spread the load over several machines, or you measure the load generator, not the broker.
• Average latency. Broker latency has a long tail. Look at p95 and p99.

A worked example

In Spitfire, every VU keeps its own MQTT client, just like a device: it connects on the first step and stays connected until the VU stops. Connecting is not part of the step's duration, but a failed connect fails the step; if the connection drops, the VU reconnects on its next step and renews its subscriptions. The client id is a template, spitfire-{{__VU}} by default, and the VU number is unique across every scenario and runner of a run, so the same-client-id trap is closed by default.

This test connects 500 devices over 2 minutes; for 10 minutes each sends telemetry at QoS 1 every 4–6 seconds (about 100 messages a second), while two listeners watch every device's topic:

{
  "name": "MQTT: device telemetry",
  "scenarios": [
    { "name": "devices",
      "executor": { "type": "ramping-vus", "startVUs": 0,
        "stages": [ { "duration": "2m", "target": 500 }, { "duration": "10m", "target": 500 },
                    { "duration": "30s", "target": 0 } ] },
      "steps": [ { "id": "telemetry", "name": "Send telemetry", "protocol": "mqtt", "connection": "broker",
        "mqtt": { "action": "publish", "topic": "devices/{{__VU}}/telemetry", "qos": 1,
                  "payload": "{\"device\":{{__VU}},\"temp\":{{$randInt 18 30}},\"ts\":{{$timestamp}}}" },
        "thinkTime": { "min": "4s", "max": "6s" } } ] },
    { "name": "listeners",
      "executor": { "type": "constant-vus", "vus": 2, "duration": "12m30s" },
      "steps": [ { "id": "listen", "name": "Listen to telemetry", "protocol": "mqtt", "connection": "broker",
        "mqtt": { "action": "subscribe", "topic": "devices/+/telemetry", "qos": 1, "wait": "5s" },
        "checks": [ { "type": "jsonPath", "path": "$.temp", "op": "exists" } ] } ] }
  ],
  "thresholds": [
    { "metric": "req_duration", "filter": { "step": "telemetry" }, "expr": "p(95)<100" },
    { "metric": "req_failed", "expr": "rate<0.001" }
  ]
}

The run fails if the PUBACK p95 goes over 100 ms or more than 0.1% of requests fail. The listeners' checks confirm the shape of incoming messages. A note on the subscribe step: its duration is the wait for the next message, not the time from publish to receipt; measuring that across two machines runs into the same clock problem I wrote about for Kafka.

There is also a request action for request-reply over MQTT: the VU subscribes to a reply topic (give each VU its own, such as replies/{{__VU}}), publishes, and measures the round trip to the first reply. With the connection set to MQTT 5, the request carries a response topic and correlation data, and only the reply with the matching correlation data counts; user properties, content type and message expiry can be set too, and a reply's user properties reach checks as headers.

Spitfire installs on Docker or Kubernetes with one command and has a free edition. The full guide, with connection settings, TLS and spreading thousands of devices over several runners, is here: MQTT load testing — Spitfire.

---

Drafted with AI assistance and reviewed by the author.