DEV Community: Spitfire

MQTT load testing: thousands of devices, reconnect storms and what QoS really measures

Spitfire — Wed, 14 Oct 2026 05:48:30 +0000

What stresses an MQTT broker is often not the message rate but the number of connections. Ten thousand devices, each sending a small message every few seconds, means ten thousand sessions, subscriptions and keep-alives the broker holds in memory. A broker that handles 5,000 messages a second from 20 connections without blinking can fall over at a tenth of that rate from 10,000 devices.

An MQTT load test answers three questions:

How many concurrent devices does the broker accept?
How long does a message's acknowledgement take at that load?
What happens when every device reconnects at once after an outage?

Disclosure: I build Spitfire, a self-hosted load testing tool. The method below applies to any tool; the last section shows how Spitfire does it.

What to measure

Concurrent connections: clients connected at once. Each one costs memory, a file descriptor and keep-alive traffic; the limit often comes here before CPU.
Connect rate: new connections accepted per second. The TLS handshake and authentication are expensive. Thousands of devices reconnecting together after a network outage (a reconnect storm) is a bottleneck that a steady load never shows.
Publish latency: it depends on QoS, so use the QoS your devices really use.
Fan-out: a message is copied to every matching subscriber. A few services subscribed to a broad filter such as devices/+/telemetry multiply what the broker sends.
Errors: refused connections, authentication failures, timeouts and sessions the broker closes.

What QoS really measures

"Publish latency" means something different at each QoS level:

QoS 0: the client writes the message to the socket and moves on. There is no answer from the broker, so the number you get is how fast your own client writes, not how fast the broker accepts.
QoS 1: the time to the broker's PUBACK. This is the one most IoT fleets care about.
QoS 2: the time to PUBCOMP, at the end of a four-step handshake. Expect it to be clearly slower than QoS 1, and expect the broker to hold more state per in-flight message.

Testing at QoS 0 because "it is faster" and then running production at QoS 1 gives you a number that describes neither.

Shaping the load

The natural model is one virtual user per device: each VU opens its own connection, sends a message at intervals and stays connected. The message rate follows from the device count and the interval: 500 devices sending every 5 seconds is 100 messages a second.

Give the interval a little randomness (say 4–6 s) instead of a fixed value. Otherwise every device sends at the same moment and you get waves that do not happen in reality.

The ramp-up sets the connect rate. Reaching 500 devices in 2 minutes is about 4 connections a second; reaching the same number in 10 seconds is a reconnect storm. Run them as separate tests: one shows the broker under steady load, the other how it recovers after an outage. To find the device count where the broker breaks, raise it in steps.

Common mistakes

The same client id. A second client connecting with the same client id drops the first one's session. Give every device its own id, and if two tests hit the same broker at once, give their ids different prefixes. Otherwise you are testing session takeover, endlessly.
Many messages from few connections. 10,000 messages a second over 10 connections stresses the broker very differently from ten thousand devices. Keep the device count close to reality and set the rate with the interval.
Retained messages and persistent sessions. Messages sent with retain, and sessions without clean session, stay on the broker after the test. Use a separate topic tree for the test and plan the cleanup.
The load generator's own limit. Every connection is a socket on the generating machine too. For thousands of devices raise the file descriptor limit and spread the load over several machines, or you measure the load generator, not the broker.
Average latency. Broker latency has a long tail. Look at p95 and p99.

A worked example

In Spitfire, every VU keeps its own MQTT client, just like a device: it connects on the first step and stays connected until the VU stops. Connecting is not part of the step's duration, but a failed connect fails the step; if the connection drops, the VU reconnects on its next step and renews its subscriptions. The client id is a template, spitfire-{{__VU}} by default, and the VU number is unique across every scenario and runner of a run, so the same-client-id trap is closed by default.

This test connects 500 devices over 2 minutes; for 10 minutes each sends telemetry at QoS 1 every 4–6 seconds (about 100 messages a second), while two listeners watch every device's topic:

{
  "name": "MQTT: device telemetry",
  "scenarios": [
    { "name": "devices",
      "executor": { "type": "ramping-vus", "startVUs": 0,
        "stages": [ { "duration": "2m", "target": 500 }, { "duration": "10m", "target": 500 },
                    { "duration": "30s", "target": 0 } ] },
      "steps": [ { "id": "telemetry", "name": "Send telemetry", "protocol": "mqtt", "connection": "broker",
        "mqtt": { "action": "publish", "topic": "devices/{{__VU}}/telemetry", "qos": 1,
                  "payload": "{\"device\":{{__VU}},\"temp\":{{$randInt 18 30}},\"ts\":{{$timestamp}}}" },
        "thinkTime": { "min": "4s", "max": "6s" } } ] },
    { "name": "listeners",
      "executor": { "type": "constant-vus", "vus": 2, "duration": "12m30s" },
      "steps": [ { "id": "listen", "name": "Listen to telemetry", "protocol": "mqtt", "connection": "broker",
        "mqtt": { "action": "subscribe", "topic": "devices/+/telemetry", "qos": 1, "wait": "5s" },
        "checks": [ { "type": "jsonPath", "path": "$.temp", "op": "exists" } ] } ] }
  ],
  "thresholds": [
    { "metric": "req_duration", "filter": { "step": "telemetry" }, "expr": "p(95)<100" },
    { "metric": "req_failed", "expr": "rate<0.001" }
  ]
}

The run fails if the PUBACK p95 goes over 100 ms or more than 0.1% of requests fail. The listeners' checks confirm the shape of incoming messages. A note on the subscribe step: its duration is the wait for the next message, not the time from publish to receipt; measuring that across two machines runs into the same clock problem I wrote about for Kafka.

There is also a request action for request-reply over MQTT: the VU subscribes to a reply topic (give each VU its own, such as replies/{{__VU}}), publishes, and measures the round trip to the first reply. With the connection set to MQTT 5, the request carries a response topic and correlation data, and only the reply with the matching correlation data counts; user properties, content type and message expiry can be set too, and a reply's user properties reach checks as headers.

Spitfire installs on Docker or Kubernetes with one command and has a free edition. The full guide, with connection settings, TLS and spreading thousands of devices over several runners, is here: MQTT load testing — Spitfire.

Drafted with AI assistance and reviewed by the author.

Load testing your database with your real query mix: from pg_stat_statements to a weighted load test

Spitfire — Thu, 08 Oct 2026 11:49:50 +0000

Most database load tests I have seen run one or two queries in a loop. That tells you how fast those queries are when nothing else is happening. It does not tell you what your database does on a normal Tuesday, when one lookup is called 120,000 times an hour, a customer listing 45,000 times and a report 400 times, all competing for the same buffer cache, the same connections and the same locks.

The good news: your database already knows that profile. This post shows how to export it and turn it into a weighted load test.

1. Export the query profile

PostgreSQL (the pg_stat_statements extension must be enabled), from psql:

\copy (SELECT query, calls, total_exec_time, rows
  FROM pg_stat_statements
  WHERE dbid = (SELECT oid FROM pg_database WHERE datname = current_database())
  ORDER BY calls DESC LIMIT 200) TO 'stats.csv' WITH CSV HEADER

MySQL / MariaDB, save the result as CSV or TSV with its header row (mysql --batch gives TSV):

SELECT DIGEST_TEXT, COUNT_STAR, SUM_TIMER_WAIT
  FROM performance_schema.events_statements_summary_by_digest
  WHERE SCHEMA_NAME = DATABASE() AND DIGEST_TEXT IS NOT NULL
  ORDER BY COUNT_STAR DESC LIMIT 200;

A few things to know about this data before you use it:

The queries are normalized. Constants are replaced by placeholders ($1 in PostgreSQL, ? in MySQL). To replay them you have to give each placeholder a value: ids from a CSV of real keys are best; random values in a realistic range are the next best thing. Never replay one constant: that measures the cache, not the database.
Some rows are noise. BEGIN, COMMIT, SET and the statistics query itself show up. Drop them.
Some rows are writes. Decide deliberately whether a write belongs in the test, and never run one against production.
Some texts are lossy. MySQL collapses IN (1, 2, 3) to IN (...), and long statements get cut. Those need a manual edit.
Call counts are your weights. You do not need to convert them to percentages; relative weights do the job.

2. Turn it into a weighted mix

The idea is simple: in each iteration, a virtual user picks one query at random, with probability proportional to its call count. Over a few minutes the database sees the same proportions it sees in production.

In Spitfire (from 0.18.0), a SQL step can hold such a mix directly:

{ "id": "orders", "name": "Order queries", "protocol": "sql", "connection": "orders-db",
  "sql": { "mix": [
    { "name": "order_by_id", "query": "SELECT id, status, total FROM orders WHERE id = $1",
      "params": ["{{$randInt 1 100000}}"], "weight": 120000 },
    { "name": "customer_orders", "query": "SELECT id, status, total, created_at FROM orders WHERE customer_id = $1 ORDER BY created_at DESC LIMIT 20",
      "params": ["{{$randInt 1 20000}}"], "weight": 45000 },
    { "name": "pending_count", "query": "SELECT count(*) FROM orders WHERE status = $1",
      "params": ["pending"], "weight": 6000 },
    { "name": "hourly_summary", "query": "SELECT date_trunc('hour', created_at) AS h, count(*), sum(total) FROM orders WHERE created_at > now() - interval '1 day' GROUP BY 1 ORDER BY 1",
      "weight": 400 }
  ] } }

Here order_by_id runs in about 70% of iterations and hourly_summary in about 0.2%. A step holds up to 200 queries.

You do not have to write that JSON by hand. In the editor, Bulk add / import takes the CSV/TSV export from step 1 (or, for admins, reads the statistics straight through the step's connection: only the statistics query, top N by calls, in a read-only transaction with a timeout). The review table shows each statement with its call count and its kind (read, write, lock, INTO, procedure), skips transaction and session commands, leaves writes unticked, and lets you bind every placeholder to a data column or a fake value. The same dialog also splits a pasted script or a .sql file, dialect-aware (PostgreSQL dollar quotes, MySQL DELIMITER, SQL Server GO, Oracle /), and turns repeated statements into weights.

3. Look at each query, not just the total

A mixed step has one latency distribution, and that is exactly the problem: a step p95 of 8 ms can hide a report query whose p95 went from 300 ms to 2 s, because it is 0.2% of the calls.

So measure each query on its own. Spitfire reports each query of a mix as its own series (sql_query_duration, sql_query_failed), and the run page and the HTML/PDF report list calls, share, p50, p95, p99 and error rate per query. Thresholds can target a single query by name:

"thresholds": [
  { "metric": "sql_query_duration", "filter": { "check": "order_by_id" }, "expr": "p(95)<20" },
  { "metric": "sql_query_duration", "filter": { "check": "customer_orders" }, "expr": "p(95)<50" },
  { "metric": "sql_query_duration", "filter": { "check": "hourly_summary" }, "expr": "p(95)<500" },
  { "metric": "req_failed", "expr": "rate<0.001" }
]

That makes the test usable as a gate in CI: the run fails when a specific query regresses, and the output names it.

4. Keep reads read-only

Replaying production queries against a database is only comfortable if you are sure nothing writes. A keyword search for INSERT is not enough: writes hide in CTE bodies (WITH d AS (DELETE … RETURNING *) SELECT …), behind EXPLAIN ANALYZE, and in row locks (FOR UPDATE, LOCK IN SHARE MODE, WITH (UPDLOCK)), while a harmless replace() call or a column named update looks like a write to a naive check.

Spitfire tokenizes each statement per dialect and counts keywords only where a statement starts (the beginning, a CTE body, after a CTE list or EXPLAIN, a subquery). On PostgreSQL and MySQL, reads also run in a read-only transaction that is rolled back, so the database itself refuses hidden writes such as a function with side effects. SQL Server and Oracle have no such transaction, so there only the statement check protects the data, and you should connect with a read-only database user. Writes need an explicit approval on the step and a confirmation every time the test starts.

5. Practical checklist

Run against a copy of production with production-like data volume and distribution. A small table fits in memory and every plan looks great.
Reset or snapshot pg_stat_statements (pg_stat_statements_reset()) before a representative window, so the counts describe a normal day and not last month's migration.
Size concurrency from the application's real pool sizes (6 instances × 20 connections = 120 concurrent queries at most), and remember that the load generator's own pools add up across machines.
Ramp up in steps and watch where p99 bends, not just where errors start.
Read the database's own metrics (connections, lock waits, cache hit ratio, replica lag) alongside the load stages.

Spitfire is self-hosted: it runs on your own servers, so the query texts and statistics you import stay in your network. The full guide, with the single-query basics and the complete example test, is here: Database load testing: PostgreSQL, MySQL, SQL Server and Oracle.

Drafted with AI assistance and reviewed by the author.

Kafka load testing: end-to-end latency, consumer lag and the clock trap

Spitfire — Tue, 06 Oct 2026 21:25:53 +0000

In a system built on Kafka, the bottleneck is often not the HTTP layer but the message path: producers slow down waiting for the broker's acknowledgement, consumers fall behind, and consumer lag grows. A Kafka load test pushes that path at a realistic rate and answers three questions:

At what latency do the brokers accept the throughput you need?
Can the consumers read at the same rate?
How does the number of messages left behind change as the load grows?

Disclosure: I build Spitfire, a self-hosted load testing tool. The method below applies to any tool; the last section shows how Spitfire does it.

What to measure

Producer latency: the time from sending a message to the broker's acknowledgement. It depends on acks: acks=all waits for every in-sync replica and is the slowest, acks=1 waits for the leader only, acks=0 does not wait and leaves nothing to measure. Use your production setting.
End-to-end latency: the time from producing a record to consuming it. This is the delay users actually feel. While the producer's ack time stays short, this one grows as consumers fall behind.
Throughput: messages per second and, with the message size, bytes per second. 10,000 messages a second at 1 KB and the same count at 100 KB are very different loads.
Consumer lag: the gap between the last offset a consumer group has committed in a partition and the end of that partition, in other words the messages not processed yet. A small, steady lag is normal; a lag that keeps growing under load means the consumers cannot keep up.
Errors: rejected messages, timeouts, authorization failures and interruptions during a rebalance.

The clock trap

The obvious way to measure end-to-end latency is to put a timestamp in the message and subtract it from the time the consumer receives it. If the producer and the consumer run on different machines, that number is wrong: two machines' clocks can differ by more than the latency you are trying to measure. Even with NTP, a few milliseconds of drift is normal, and your "p95 end-to-end latency" ends up showing the clock difference, not your system.

Only compare times read from the same clock: produce and consume the same record on the same machine (and the same process, using a monotonic clock), and accept that you measure a sample of the records rather than all of them.

Shaping the load

On the producer side the target is usually a rate ("5,000 order events a second"), not a number of users. So instead of a fixed number of virtual users, use a constant arrival rate: the message rate does not drop when the broker slows down, and the latency shows as it is. In a closed model (fixed VUs), producers slow down with the broker and the problem hides as a quiet drop in throughput.

On the consumer side the count matters: a consumer group runs at most as many active consumers as there are partitions. Put 20 consumers on a topic with 12 partitions and 8 of them sit idle. Match the consumer count to your production instances.

Common mistakes

Unrealistic messages. Sending the same small message every time flatters compression and caches. Produce messages whose size and content resemble production and change every time.
Key distribution. Messages go to a partition by key. Few keys pile the load onto a few partitions (a hot partition); an empty key lets the partitioner spread messages.
A shared topic. Do not write test messages to a topic that production consumers read. Use a separate topic and consumer group, and plan retention and cleanup up front.
Average latency. Broker latency has a long tail: log segment rolls, replica sync and GC pauses cause rare but large spikes. Look at p95 and p99, not the average.
A short test. To see whether lag grows, the load has to stay steady. Leave a ramp-up of a few minutes and a steady part of at least 10–15 minutes.

Watching your real service's lag

The most useful lag in a load test is often not the test's own consumer group but your real service's group: does the service that processes these events fall behind under the load you generate, and by how much?

You can read that without touching the service: Kafka's admin API lists a partition's end offset and a group's committed offset, and lag is the difference. The load test never joins the group, commits or resets offsets, so watching it does not change how the service works. The principal needs Describe on the topic and on the group.

With Spitfire

In Spitfire a Kafka test is built in a web editor or written as JSON. The produce step waits for the broker's acknowledgement (no linger), and the VUs of a runner share one producer client, like a real service. The consume step gives every VU its own consumer client; with a groupId they share the partitions like real consumer instances.

For the two measurements above:

Every produced record gets a spitfire-ts header. When a consume step reads a record stamped in the same run on the same runner, it records kafka_e2e_latency. Spitfire never compares two machines' clocks; across N runners about 1/N of the records are measured, and the run page says so.
A consume step with a fixed group watches that group's lag, and any Kafka step can watch other groups with lagGroups, read-only, every 2 seconds, as kafka_consumer_lag.

This test produces 500 order events a second for 15 minutes while 10 consumers read the topic, and watches the real order-service group:

{
  "name": "Kafka: produce and consume order events",
  "scenarios": [
    { "name": "producer",
      "executor": { "type": "constant-arrival-rate", "rate": 500, "timeUnit": "1s",
                    "duration": "15m", "preAllocatedVUs": 20, "maxVUs": 100 },
      "steps": [ { "id": "produce", "name": "Produce order event", "protocol": "kafka", "connection": "kafka",
        "kafka": { "action": "produce", "topic": "orders", "key": "order-{{$uuid}}",
          "value": "{\"orderId\":\"{{$uuid}}\",\"amount\":{{$randInt 10 5000}}}",
          "lagGroups": [ "order-service" ] } } ] },
    { "name": "consumer",
      "executor": { "type": "constant-vus", "vus": 10, "duration": "15m" },
      "steps": [ { "id": "consume", "name": "Consume order event", "protocol": "kafka", "connection": "kafka",
        "kafka": { "action": "consume", "topic": "orders", "groupId": "spitfire-loadtest", "wait": "5s" },
        "checks": [ { "type": "jsonPath", "path": "$.orderId", "op": "exists" } ] } ] }
  ],
  "thresholds": [
    { "metric": "req_duration", "filter": { "step": "produce" }, "expr": "p(95)<50" },
    { "metric": "req_failed", "expr": "rate<0.001" },
    { "metric": "kafka_e2e_latency", "expr": "p(95)<500" },
    { "metric": "kafka_consumer_lag", "filter": { "check": "order-service" }, "expr": "value<100" }
  ]
}

The run fails if the produce acknowledgement's p95 goes over 50 ms, more than 0.1% of requests fail, end-to-end p95 goes over 500 ms, or the service's group is left more than 100 records behind when the run ends.

Spitfire installs on Docker or Kubernetes with one command and has a free edition. The full guide, with permissions, per-partition lag and how this lines up with your broker metrics, is here: Kafka load testing — Spitfire.