DEV Community

Smrati
Smrati

Posted on

How to process 1 million IoT events per day without breaking your backend

#iot #backend #kafka #assettracking

1 million events per day seems impressive until you break it down — that is only 11.5 events per second. A cluster of 500 GPS devices updating their positions every 43 seconds would reach the desired level of performance easily enough.
Here's what kind of architecture could handle this load without breaking a sweat.

What kills naive architectures in IoT backend development

The first idea one comes up with when designing an IoT backend system architecture typically goes like this: device posts message to API → Express receives the event and puts into the Postgres database → mission accomplished. It scales well until you start talking about more than 10 devices — say, 500 with a new update every 30 seconds. Your poor Express process and your barely optimized PostgreSQL will hardly manage to keep up.

Scalable architecture

Step 1

MQTT broker — device messages get sent to either Mosquitto or HiveMQ. Persistent connection, no overhead per each message handshake.

Step 2

Message queue based on Kafka — MQTT broker forwards all events to the queue via the Kafka MQTT bridge. Kafka handles sudden spikes, has replay, decouples consumption and processing entirely.

Step 3

Consumers/worker nodes — consumers based on Node.js or Python consume from Kafka asynchronously. Each consumer handles the enrichment, validation, and alerts separately.

Step 4

TimescaleDB + Redis — workers store data in TimescaleDB for storage, update Redis state for the live stream. Dashboards consume from Redis, but not from write streams.

Step 5

WebSocket gateway — separate system consumes live state from Redis and forwards updates to dashboards using WebSocket. Separate from the consumption pipeline.

Step 1 – MQTT to Kafka bridge

The MQTT to Kafka bridge listens to all device topics and sends all messages to Kafka in a single connection. Be lightweight and efficient:

const mqtt = require('mqtt')
const { Kafka } = require('kafkajs')

const kafka = new Kafka({ brokers: ['kafka:9092'] })
const producer = kafka.producer()
await producer.connect()

const mqttClient = mqtt.connect('mqtt://broker:1883')
mqttClient.subscribe('assets/+/telemetry')

mqttClient.on('message', async (topic, payload) => {
  await producer.send({
    topic: 'iot-events',
    messages: [{ value: payload }]
  })
  // Fire and forget — no blocking, no await cascade
})
Enter fullscreen mode Exit fullscreen mode

Step 2 – Kafka consumer with batch writes

The Kafka consumer pulls events in batches and writes them to the write database in batch mode. Do not perform single row inserts as they have a horrible impact on performance. Batching everything is key:

const consumer = kafka.consumer({ groupId: 'iot-processor' })
await consumer.connect()
await consumer.subscribe({ topic: 'iot-events' })

await consumer.run({
  eachBatch: async ({ batch }) => {
    const rows = batch.messages.map(m => {
      const d = JSON.parse(m.value.toString())
      return [d.assetId, d.lat, d.lng, d.tempC, new Date()]
    })

    // Single bulk INSERT for entire batch
    await db.query(
      `INSERT INTO asset_locations
       (asset_id,lat,lng,temp_c,time) VALUES %L`,
      [rows]
    )

    // Update Redis live state per asset
    for (const row of rows) {
      await redis.set(`twin:${row[0]}`, JSON.stringify(row))
    }
  }
})
Enter fullscreen mode Exit fullscreen mode

Step 3 – Redis for live data, TimescaleDB for history

Here is where most teams go wrong. Never use a write database for live queries — it's a cardinal sin. Have Redis handle real-time state queries (sub-millisecond read times) while TimescaleDB handles time series history. There will be zero contention between the two:

// Dashboard API — reads from Redis, never touches TimescaleDB
app.get('/assets/live', async (req, res) => {
  const keys = await redis.keys('twin:*')
  const states = await Promise.all(
    keys.map(k => redis.get(k).then(JSON.parse))
  )
  res.json(states)
})

// Historical query — goes to TimescaleDB
app.get('/assets/:id/history', async (req, res) => {
  const rows = await db.query(
    `SELECT * FROM asset_locations
     WHERE asset_id=$1 AND time > NOW()-INTERVAL '24h'`,
    [req.params.id]
  )
  res.json(rows)
})
Enter fullscreen mode Exit fullscreen mode

Backpressure tip

: If the Kafka consumer is lagging behind — slow batch inserts vs. incoming ingest rate — then don't simply throw more consumers at the problem. Check whether the bulk insert size (preferably 500+ records per insert). Also enable Redis pipeline mode for multi set operations. In my experience, these simple changes give 3-5x improvement over horizontal scaling.

Numbers - this is what this solves

  • 11.5/s average events per second at 1M/day
  • 50k+ events/sec Kafka handles with just one broker
  • <1ms Redis live state read latency
  • 10M+ rows/day TimescaleDB can easily handle

Suggested tech stack

  1. Mosquitto / HiveMQ
  2. Kafka + KafkaJS
  3. Node.js workers
  4. Redis
  5. TimescaleDB
  6. Docker Compose

Setup the entire tech stack locally with Docker Compose for testing - there are official Docker images available for Kafka, Zookeeper, Redis, and TimescaleDB. In production, hosted Kafka (Confluent Cloud or MSK) eliminates much of the operational overhead. The application logic does not change.

Are you building a system for managing millions of events coming from IoT devices? AssetTrackPro takes care of this for you, allowing you to focus on the application logic. Learn more →

Building an IoT backend? AssetTrackPro provides out-of-the-box integration for Kafka, Redis, and TimescaleDB - you can start ingesting telemetric data immediately without setting up the whole infrastructure yourself.

See AssetTrackPro in action→

Top comments (0)