DEV Community: Supun Sriyananda

Python vs C++ for Embedded Systems: When to Use Each

Supun Sriyananda — Fri, 10 Jul 2026 07:03:04 +0000

When you first step into the world of embedded systems, one of the earliest and most consequential decisions you will face is choosing a programming language. Two names come up more than any others: Python and C++. Both are powerful, both have passionate communities, and both are genuinely useful — but for very different reasons and in very different contexts. This article is not about declaring a winner. It is about understanding why each language exists in this space, what trade-offs you are actually making, and how to make a confident, informed decision for your next project.

Understanding the Fundamental Difference

Before comparing features, it helps to understand why these two languages feel so different at a deeper level.

C++ is a compiled, statically-typed, systems-level language. When you write C++, you are writing code that gets translated directly into machine instructions. You manage memory manually. You control exactly when objects are created and destroyed. The hardware does precisely what you tell it to, nothing more and nothing less. This directness is both its superpower and its source of complexity.

Python, by contrast, is an interpreted, dynamically-typed, high-level language. A Python runtime sits between your code and the hardware, managing memory automatically through garbage collection, resolving types at runtime, and handling a lot of bookkeeping so you don't have to. This makes Python wonderfully expressive and fast to write, but it introduces overhead that matters enormously on constrained hardware.

The mental model to hold onto is this: C++ gives you control, Python gives you speed of development. Both are valuable. The question is which one your project needs more.

Where C++ Shines in Embedded Systems

1. Bare-Metal and Resource-Constrained Environments

If you are programming a microcontroller like an STM32, an AVR ATmega, or an ESP32 running its native SDK, C++ is almost always your primary language. These devices often have kilobytes — not megabytes — of RAM, and they have no operating system to fall back on. Python's runtime alone would consume more memory than the entire chip provides.

// C++ on a bare-metal microcontroller (e.g., STM32 HAL)
// Direct register manipulation gives you precise, deterministic control
// over hardware peripherals with zero overhead

#include "stm32f4xx_hal.h"

// A simple GPIO toggle to blink an LED — every cycle counts here
void blink_led(GPIO_TypeDef* port, uint16_t pin, uint32_t delay_ms) {
    while (true) {
        HAL_GPIO_TogglePin(port, pin);   // Direct hardware write
        HAL_Delay(delay_ms);              // Blocking delay — acceptable on bare metal
    }
}

The key insight here is determinism. In safety-critical or real-time systems — think motor controllers, medical devices, or industrial sensors — you need to guarantee that a particular piece of code executes within a precise time window. Python's garbage collector can pause your program at unpredictable moments. C++ has no such hidden pauses.

2. Real-Time Operating Systems (RTOS)

When you step up to an RTOS like FreeRTOS, Zephyr, or ThreadX, C++ remains the dominant choice. RTOS kernels are themselves written in C/C++, and the APIs they expose are designed for direct use from those languages. You get preemptive multitasking, precise interrupt handling, and microsecond-level timing — all with the kind of control that C++ makes natural.

// FreeRTOS task written in C++ — structured, type-safe, and efficient
// Each task runs as an independent thread with its own stack allocation

#include "FreeRTOS.h"
#include "task.h"

class SensorTask {
public:
    // Static wrapper needed because FreeRTOS expects a plain C function pointer
    static void taskEntry(void* param) {
        // Cast back to our class instance — a common C++/RTOS pattern
        static_cast<SensorTask*>(param)->run();
    }

    void start() {
        // Stack size (512 words) and priority are explicit — you own these decisions
        xTaskCreate(taskEntry, "Sensor", 512, this, tskIDLE_PRIORITY + 1, &handle_);
    }

private:
    TaskHandle_t handle_;

    void run() {
        while (true) {
            readSensor();
            vTaskDelay(pdMS_TO_TICKS(100)); // Yield for exactly 100ms
        }
    }

    void readSensor() {
        // Hardware-specific sensor reading logic lives here
    }
};

3. Performance-Critical Signal Processing

Digital signal processing (DSP), motor control algorithms, PID loops, and image processing pipelines all benefit enormously from C++. The language lets you write tightly optimized loops, use SIMD intrinsics, and avoid any dynamic allocation in hot paths. When you need to process a sensor stream at 10 kHz with a 50-microsecond deadline, Python is not a practical option.

4. Production Firmware

If you are shipping a product — a consumer device, an industrial controller, a wearable — your firmware will almost certainly be C or C++. It compiles to a fixed binary, has predictable resource usage, can be certified against safety standards, and does not require a runtime environment on the target device.

Where Python Shines in Embedded Systems

1. Rapid Prototyping on Linux-Based Boards

Single-board computers like the Raspberry Pi, BeagleBone, or NVIDIA Jetson run full Linux, which means they can run a complete Python interpreter. On these platforms, Python becomes extraordinarily powerful for prototyping. You can wire up a sensor, write twenty lines of Python, and have a working proof-of-concept in under an hour.

# Python on Raspberry Pi using the RPi.GPIO library
# This simplicity is Python's greatest asset for hardware prototyping

import RPi.GPIO as GPIO
import time

LED_PIN = 18  # BCM numbering

GPIO.setmode(GPIO.BCM)
GPIO.setup(LED_PIN, GPIO.OUT)

# A PWM-based fade effect — would take far more boilerplate in C++
pwm = GPIO.PWM(LED_PIN, 100)  # 100 Hz frequency
pwm.start(0)

try:
    while True:
        # Gradually increase and decrease brightness
        for duty_cycle in range(0, 101, 5):
            pwm.ChangeDutyCycle(duty_cycle)
            time.sleep(0.05)
        for duty_cycle in range(100, -1, -5):
            pwm.ChangeDutyCycle(duty_cycle)
            time.sleep(0.05)
finally:
    pwm.stop()
    GPIO.cleanup()  # Always clean up GPIO state on exit

2. MicroPython and CircuitPython on Microcontrollers

This is where the story gets more nuanced. MicroPython and Adafruit's CircuitPython are lean Python implementations designed specifically to run on microcontrollers like the RP2040, ESP32, and STM32. They bring Python's expressiveness to hardware with just a few hundred kilobytes of flash.

These environments are excellent for education, hobbyist projects, and rapid iteration. The trade-off is that you are running an interpreter with managed memory, so real-time guarantees are off the table and performance is a fraction of native C++.

# MicroPython on an RP2040 (Raspberry Pi Pico)
# The simplicity here is remarkable — I2C in just a few lines

from machine import I2C, Pin
import time

# Initialize I2C bus on pins 4 (SDA) and 5 (SCL)
i2c = I2C(0, sda=Pin(4), scl=Pin(5), freq=400_000)

# Scan for connected devices — invaluable during prototyping
devices = i2c.scan()
print(f"Found I2C devices at addresses: {[hex(d) for d in devices]}")

# Read 6 bytes from a hypothetical sensor at address 0x68
if 0x68 in devices:
    data = i2c.readfrom(0x68, 6)
    # Parse raw bytes — still manageable in Python
    accel_x = int.from_bytes(data[0:2], 'big')
    print(f"Accelerometer X: {accel_x}")

3. Data Pipelines, ML Inference, and High-Level Logic

On Linux-based edge devices, Python's ecosystem is unmatched. Libraries like NumPy, OpenCV, TensorFlow Lite, and PySerial let you build sophisticated data pipelines, run machine learning inference, and communicate with sensors — all without writing a single line of C++. This is where Python earns its place even in serious embedded projects.

4. Testing, Tooling, and Automation

Even on projects where the final firmware is C++, Python plays a huge role in the surrounding ecosystem. Test frameworks, hardware-in-the-loop test scripts, build automation, log parsers, and configuration generators are all commonly written in Python. It is the glue language of the embedded world.

A Direct Comparison Across Key Dimensions

Performance

C++ wins decisively here. A well-written C++ program will typically run 10x to 100x faster than equivalent Python, and in tight real-time loops the gap is even larger. On microcontrollers without a Python runtime, C++ is often the only option.

Development Speed

Python wins just as decisively. The combination of dynamic typing, interactive REPL, extensive libraries, and concise syntax means you can explore ideas, test hypotheses, and build prototypes dramatically faster in Python than in C++.

Memory Usage

C++ gives you granular control. You can preallocate fixed-size buffers, use stack memory exclusively in critical sections, and tune every byte of usage. Python's runtime itself consumes significant memory before your application even starts — typically several megabytes for CPython, several hundred kilobytes for MicroPython.

Ecosystem and Libraries

For general-purpose computing tasks (networking, data science, ML, web APIs), Python's ecosystem is simply enormous. For low-level hardware interaction, peripheral drivers, and embedded frameworks, the C/C++ ecosystem has decades of depth.

Debugging and Observability

Python is easier to debug interactively. You can print, inspect, and modify state at runtime with minimal friction. C++ debugging on embedded targets requires more tooling — JTAG/SWD debuggers, GDB, logic analyzers — but that tooling is very mature and gives you deep visibility into the hardware.

Safety and Reliability

C++ makes it possible (with discipline) to write code with no dynamic allocation, bounded execution time, and no hidden runtime behavior. This is essential for safety-critical systems. Python's garbage collector and dynamic nature make such guarantees essentially impossible.

The Hybrid Approach: Using Both

In practice, many serious embedded projects use both languages in complementary roles. A common architecture looks like this:

The C++ layer handles all time-critical operations: sensor interrupt handlers, motor control loops, communication protocol drivers, and safety watchdogs. This layer is compiled to a binary that runs deterministically on the target hardware.

The Python layer sits above it, handling high-level orchestration: reading data from the C++ layer over a serial or USB interface, applying machine learning models, sending data to cloud services, and providing a configuration interface. On a Raspberry Pi, this layer might coordinate with a microcontroller over UART while simultaneously pushing data to an MQTT broker.

# Python orchestration layer on a Raspberry Pi
# Talking to a C++ firmware over serial — a very common real-world pattern

import serial
import struct
import json

class FirmwareBridge:
    """
    High-level Python wrapper around low-level C++ firmware.
    The firmware handles real-time control; we handle logic and connectivity.
    """
    def __init__(self, port: str, baud: int = 115200):
        self.serial = serial.Serial(port, baud, timeout=1.0)

    def read_sensor_packet(self) -> dict | None:
        """
        Read a binary packet from firmware.
        The C++ side sends fixed-width structs for efficiency.
        """
        header = self.serial.read(1)
        if not header or header[0] != 0xAA:  # Sync byte
            return None

        # Unpack a 3-float struct: temperature, humidity, pressure
        raw = self.serial.read(12)
        if len(raw) < 12:
            return None

        temp, humidity, pressure = struct.unpack('<fff', raw)
        return {"temperature": temp, "humidity": humidity, "pressure": pressure}

    def send_command(self, command_id: int, payload: bytes = b'') -> None:
        """Send a control command down to the C++ firmware."""
        packet = bytes([0xBB, command_id, len(payload)]) + payload
        self.serial.write(packet)

This hybrid model lets you play to each language's strengths: C++ for the parts where performance and determinism are non-negotiable, Python for the parts where developer velocity and ecosystem richness matter most.

Decision Framework: How to Choose

Rather than a rigid rule, think of it as a set of questions to work through for any given project or module.

Ask: Does this code have hard real-time requirements? If yes, use C++. Real-time means a missed deadline is a failure, not just a slowdown.

Ask: Does this run on a microcontroller without an OS? If yes, use C++. Python runtimes need significantly more resources than bare-metal MCUs typically provide.

Ask: Am I building a proof-of-concept or exploring hardware behavior? If yes, start with Python (on a suitable platform). Validate your idea fast before optimizing.

Ask: Does this code interact with complex libraries — ML, computer vision, networking? If yes, Python is likely more practical. The C++ equivalents exist but are far harder to integrate.

Ask: Will this code ship in a product with safety or reliability requirements? If yes, lean heavily toward C++, potentially with a formal coding standard like MISRA C++.

Ask: Is this part of a larger system's tooling, testing, or configuration? If yes, Python is almost certainly the right choice regardless of what the firmware itself uses.

Conclusion

The Python vs C++ question in embedded systems is not a rivalry — it is a spectrum. C++ is the language of determinism, performance, and direct hardware control. Python is the language of expressiveness, rapid iteration, and ecosystem richness. Understanding why each language works the way it does is far more valuable than memorizing a comparison table.

The most effective embedded engineers are fluent in both. They reach for C++ when they need to talk directly to hardware with precision and speed, and they reach for Python when they need to move fast, analyze data, or connect their hardware to the broader software world. Knowing which tool to pick up, and why, is the skill worth developing.

Implementing Exponential Backoff: Preventing Thundering Herd Problems

Supun Sriyananda — Wed, 08 Jul 2026 08:47:10 +0000

When a broker goes down and a thousand edge devices all try to reconnect every second, the moment the broker comes back online it gets hit with a thousand simultaneous connection requests and dies again. This is the thundering herd problem, and it is entirely self-inflicted. The fix is exponential backoff with jitter — a reconnection strategy every networked embedded application should implement from day one.

🔗 See project on GitHub: Resilient Edge MQTT Client

The Wrong Way

# Every device retries every second. 1000 devices = 1000 req/s on reconnect.
while True:
    try:
        client.connect(broker_host, broker_port)
        break
    except Exception:
        time.sleep(1)  # Fixed delay — all devices retry in lockstep

Fixed-delay retry is the problem. Every device that disconnected at the same time will retry at exactly the same time, forever, until one of them happens to succeed. At scale, this makes outages self-perpetuating.

The Right Way

import time
import random

def connect_with_backoff(client, broker_host, broker_port,
                         min_delay=1, max_delay=300):
    """
    Reconnect with exponential backoff and jitter.

    Delay sequence (approximate): 1s, 2s, 4s, 8s, 16s, 32s, 60s, 60s...
    Jitter adds ±10% randomness so devices that disconnected together
    do not retry together.
    """
    delay = min_delay

    while True:
        try:
            # Jitter is proportional to the current delay, not fixed.
            # At delay=1 it adds up to 0.1s. At delay=60 it adds up to 6s.
            # This keeps devices spread out even after long outages.
            jitter = random.uniform(0, delay * 0.1)
            time.sleep(delay + jitter)

            client.connect(broker_host, broker_port)
            return  # Success — caller resets delay to min_delay

        except Exception as e:
            print(f"Connection failed: {e}. Retrying in {delay:.1f}s")
            delay = min(delay * 2, max_delay)

The three decisions baked into this function are worth understanding individually.

Doubling the delay (the exponential part) means the load on the broker drops by half with each retry cycle. After a few rounds, devices are spread so far apart in time that the broker recovers comfortably before the next wave arrives.

Capping at max_delay prevents devices from backing off indefinitely. A cap of 60–300 seconds is usually right for embedded systems — long enough to give the broker real recovery time, short enough that you do not lose hours of data during a brief outage.

Proportional jitter is the part most implementations get wrong. Adding a small fixed jitter like random.uniform(0, 0.5) works when delay is 1 second but is meaningless when delay is 60 seconds — all devices are still retrying within the same half-second window. Making jitter proportional to the current delay (10% here) keeps devices spread out at every stage of backoff.

Three Ways to Add Jitter

The 10% proportional jitter above is deliberately conservative — it keeps retries close to the intended backoff curve while still desynchronizing devices. But it's not the only approach, and for larger fleets more aggressive strategies spread load better. These are the three established patterns, all built on the same base calculation:

temp = min(cap, base * (2 ** attempt))  # uncapped exponential value

Full Jitter picks the delay uniformly between zero and the full exponential value. This maximizes spread but sacrifices any guaranteed minimum wait — some devices will retry almost immediately.

sleep = random.uniform(0, temp)

Equal Jitter keeps half the delay fixed and randomizes the other half. This guarantees a minimum wait while still breaking synchronization — a good middle ground when you don't want devices hammering back instantly.

sleep = temp / 2 + random.uniform(0, temp / 2)

Decorrelated Jitter bases each delay on the previous sleep time rather than the attempt count, letting the backoff wander upward more smoothly. AWS's analysis found this among the most effective at minimizing total work against a recovering service.

sleep = min(cap, random.uniform(base, prev_sleep * 3))

For a small edge fleet, proportional or equal jitter is plenty. For thousands of clients hitting a shared API, full or decorrelated jitter spreads the recovery load more evenly.

Reset the Delay on Success

This is the step that gets forgotten. After a successful connection, the delay must reset to min_delay. If it stays at whatever value it reached during the outage, the next disconnection — even a brief one — will start with a 60-second wait instead of a 1-second wait.

class ResilientMQTTClient:
    def __init__(self, broker_host, broker_port):
        self.broker_host = broker_host
        self.broker_port = broker_port
        self.min_delay = 1
        self.max_delay = 300
        self.current_delay = self.min_delay  # Reset this on every successful connect

    def on_connect(self, client, userdata, flags, rc):
        if rc == 0:
            self.current_delay = self.min_delay  # Always reset here
            print("Connected — backoff delay reset to minimum")

    def reconnect(self):
        jitter = random.uniform(0, self.current_delay * 0.1)
        time.sleep(self.current_delay + jitter)

        try:
            self.client.reconnect()
        except Exception:
            self.current_delay = min(self.current_delay * 2, self.max_delay)

With Paho MQTT, the right place to reset is inside the on_connect callback, which fires on the background network thread whenever a connection is established. Resetting in the reconnection loop itself is too early — the connection might still fail after the reset.

Add a Restart Rate Limit at the System Level

Backoff handles network-level reconnection. But if your application itself is crashing and restarting — not just reconnecting — you need a second line of defence. systemd's StartLimitBurst and StartLimitIntervalSec directives cap how many times a service can restart within a given window before systemd stops trying and marks it failed.

[Service]
Restart=on-failure
RestartSec=5                  # Base delay before first restart
StartLimitIntervalSec=120     # Window for counting restarts
StartLimitBurst=5             # Give up after 5 restarts in 120 seconds

This prevents a crashing service from hammering a broker with rapid reconnections that bypass your application-level backoff entirely.

Don't Retry Blindly

Backoff controls when you retry. It says nothing about whether you should — and retrying the wrong things causes worse problems than the outage itself.

Not every error is retryable. Retry only on transient failures: connection timeouts and 5xx responses like 500 and 503, which mean the service is temporarily unwell and may recover. Do not retry client errors like 401 or 404 — a request that's unauthorized or points at something that doesn't exist will fail identically no matter how many times you send it. Retrying it just wastes cycles and pollutes your logs.

Non-idempotent operations need protection. If a retry can duplicate a side effect — charging a card, creating a record, sending a message — then a retry that succeeds after the original silently went through will corrupt your data. The standard fix is an idempotency key: the client attaches a unique token to the request, and the server uses it to recognize and deduplicate a retry of an operation it already completed. Reconnecting to a broker is naturally idempotent; posting a payment is not.

Always set an attempt limit. The max_delay cap stops any single wait from growing unbounded, but a client can still retry a capped delay forever. For request/response operations, give up after a fixed number of attempts (typically 5–7) and surface the error to the caller rather than retrying into the void. Persistent connections like MQTT are the exception — there, retrying indefinitely against max_delay is usually what you want.

Quick Reference

Enable exponential backoff on every service that connects to an external broker, API, or database — not just MQTT. Start with min_delay=1, max_delay=60 for most edge applications. Use proportional jitter (10% of current delay) for small fleets, or full/decorrelated jitter when thousands of clients share a service — never fixed jitter. Always reset the delay counter inside the successful-connection callback. Retry only transient errors (timeouts, 5xx), never client errors (4xx). Protect non-idempotent operations with an idempotency key, and cap request/response retries at 5–7 attempts. Pair application-level backoff with systemd restart rate limiting for complete coverage.

If you found this useful, drop a ❤️ or a comment — I'd love to hear how you handle reconnection in your own systems.

Why SQLite is Perfect for Edge Device Data Logging

Supun Sriyananda — Thu, 02 Jul 2026 08:57:42 +0000

Most embedded Linux projects do not need a database server. They need something that stores data reliably when the network is down, survives a power cut, does not eat RAM, and requires zero administration. SQLite does all of that. It is a single .db file on disk, it has been in production use for over twenty years, and it is already installed on virtually every Linux system you will ever deploy to. If you are currently writing sensor data to a flat file or a custom binary format, this article will show you why SQLite is almost always the better choice — and exactly how to use it correctly.

See the project on GitHub: Resilient Edge MQTT client

Enable WAL Mode Before Anything Else

The single most impactful thing you can do is switch from the default journal mode to Write-Ahead Logging. In the default mode, SQLite locks the entire database file during a write, which means a reader has to wait. WAL mode allows reads and writes to happen simultaneously, which matters enormously when you have one thread logging sensor data and another thread reading it for transmission. It also makes crash recovery safer — incomplete writes never corrupt the database.

import sqlite3

conn = sqlite3.connect('sensor_data.db')

# Do this immediately after opening — before any other operations
conn.execute("PRAGMA journal_mode=WAL")
conn.execute("PRAGMA synchronous=NORMAL")  # Safe with WAL, much faster than FULL

conn.commit()

synchronous=NORMAL is safe to use in combination with WAL mode. The default FULL mode flushes to disk on every transaction, which is slow on SD cards and eMMC storage. NORMAL flushes at critical checkpoints instead, giving you durability against crashes while being significantly faster on the kinds of storage you find in embedded devices.

Schema Design: Keep It Flat, Add the Right Indexes

Resist the urge to normalise aggressively. On an edge device you are almost always inserting one kind of data repeatedly and querying it by time range. A flat table with a proper index on the timestamp column handles both operations efficiently and keeps queries simple.

conn.execute("""
    CREATE TABLE IF NOT EXISTS sensor_readings (
        id        INTEGER PRIMARY KEY AUTOINCREMENT,
        timestamp REAL    NOT NULL,  -- Unix epoch, fractional seconds
        topic     TEXT    NOT NULL,  -- e.g. 'sensors/warehouse/temperature'
        payload   TEXT    NOT NULL,  -- JSON string
        synced    INTEGER NOT NULL DEFAULT 0  -- 0 = pending upload, 1 = done
    )
""")

# This index makes time-range queries fast and powers the sync queue
conn.execute("""
    CREATE INDEX IF NOT EXISTS idx_timestamp_synced
    ON sensor_readings(timestamp, synced)
""")

The synced column is a pattern worth adopting from the start. Edge devices often need to buffer data locally and then upload it in batches when connectivity returns. Tracking sync state in the same table keeps everything in one place and makes the query for "give me everything not yet sent" trivially simple:

SELECT * FROM sensor_readings WHERE synced = 0 ORDER BY timestamp ASC LIMIT 100

Always Use Batch Inserts

Never insert one row per commit in a logging loop. Each commit flushes a transaction to disk, and on embedded storage that is slow enough to miss readings at any meaningful sample rate. Batch your inserts inside a single transaction — the difference in throughput is dramatic.

import time
import json
from contextlib import contextmanager

@contextmanager
def batch_insert(conn, batch_size=100):
    """Context manager that accumulates rows and commits in one transaction."""
    rows = []

    def add(topic, payload):
        rows.append((time.time(), topic, json.dumps(payload)))
        if len(rows) >= batch_size:
            _flush(conn, rows)
            rows.clear()

    yield add

    if rows:  # Flush any remaining rows on exit
        _flush(conn, rows)

def _flush(conn, rows):
    conn.executemany(
        "INSERT INTO sensor_readings (timestamp, topic, payload) VALUES (?, ?, ?)",
        rows
    )
    conn.commit()


# Usage in a sensor loop
with batch_insert(conn, batch_size=50) as log:
    for reading in sensor_stream():
        log('sensors/warehouse/temperature', {
            'value': reading.temperature,
            'unit': 'celsius'
        })

executemany with a list of tuples is the correct tool here. It prepares the statement once and executes it repeatedly within a single transaction, which is both faster and safer than building SQL strings manually. Never concatenate user data or variable values into SQL strings directly — always use parameterised queries with ? placeholders.

Thread Safety: One Connection Per Thread

If your application is multi-threaded — and most real embedded applications are — do not share a single SQLite connection across threads. SQLite connections are not thread-safe by default, and sharing one between a logging thread and a sync thread will give you intermittent errors that are hard to reproduce. The cleanest approach is to use a connection pool or simply open a dedicated connection per thread.

import threading

# Thread-local storage: each thread gets its own connection automatically
_local = threading.local()

def get_connection(db_path: str) -> sqlite3.Connection:
    """Return a per-thread SQLite connection, creating it if needed."""
    if not hasattr(_local, 'conn') or _local.conn is None:
        _local.conn = sqlite3.connect(db_path, check_same_thread=False)
        _local.conn.row_factory = sqlite3.Row  # Access columns by name
        _local.conn.execute("PRAGMA journal_mode=WAL")
        _local.conn.execute("PRAGMA synchronous=NORMAL")
    return _local.conn

row_factory = sqlite3.Row is small but worth enabling. It makes query results accessible by column name (row['timestamp']) rather than index (row[0]), which makes your code substantially easier to read and maintain.

Manage Storage: Prevent the Database from Growing Forever

On a device with a 16 GB SD card, an unbounded database will eventually fill the storage and crash your application. Build a retention policy in from the start. The simplest approach is a periodic cleanup that deletes synced rows older than your retention window.

import time

def cleanup_old_records(conn, retain_days=7):
    """
    Delete synced records older than retain_days.
    Only deletes synced rows — unsynced data is preserved regardless of age,
    because it hasn't been uploaded yet and may still be needed.
    """
    cutoff = time.time() - (retain_days * 86400)

    cursor = conn.execute(
        "DELETE FROM sensor_readings WHERE synced = 1 AND timestamp < ?",
        (cutoff,)
    )
    conn.commit()

    deleted = cursor.rowcount
    if deleted > 0:
        # VACUUM reclaims the disk space freed by deletions
        # Run this infrequently — it rewrites the entire database file
        conn.execute("VACUUM")

    return deleted

Run cleanup_old_records once per day from a background thread or a systemd timer. The VACUUM command rewrites the database to reclaim the disk space freed by deletions — SQLite does not shrink the file automatically. VACUUM is expensive, so do not call it every cleanup cycle; once a day or once a week is usually fine depending on your write volume.

The Complete Pattern

Here is what the full setup looks like when these practices are combined:

import sqlite3
import threading
import time
import json

DB_PATH = '/var/lib/myapp/sensor_data.db'
_local = threading.local()

def get_db() -> sqlite3.Connection:
    if not hasattr(_local, 'conn') or _local.conn is None:
        conn = sqlite3.connect(DB_PATH, check_same_thread=False)
        conn.row_factory = sqlite3.Row
        conn.execute("PRAGMA journal_mode=WAL")
        conn.execute("PRAGMA synchronous=NORMAL")
        _local.conn = conn
    return _local.conn

def init_schema():
    conn = get_db()
    conn.execute("""
        CREATE TABLE IF NOT EXISTS sensor_readings (
            id        INTEGER PRIMARY KEY AUTOINCREMENT,
            timestamp REAL    NOT NULL,
            topic     TEXT    NOT NULL,
            payload   TEXT    NOT NULL,
            synced    INTEGER NOT NULL DEFAULT 0
        )
    """)
    conn.execute("""
        CREATE INDEX IF NOT EXISTS idx_timestamp_synced
        ON sensor_readings(timestamp, synced)
    """)
    conn.commit()

def log_reading(topic: str, payload: dict):
    conn = get_db()
    conn.execute(
        "INSERT INTO sensor_readings (timestamp, topic, payload) VALUES (?, ?, ?)",
        (time.time(), topic, json.dumps(payload))
    )
    conn.commit()

def get_pending(limit: int = 100) -> list:
    """Return unsynced readings, oldest first."""
    conn = get_db()
    return conn.execute(
        "SELECT * FROM sensor_readings WHERE synced = 0 ORDER BY timestamp ASC LIMIT ?",
        (limit,)
    ).fetchall()

def mark_synced(row_ids: list[int]):
    """Mark a batch of rows as successfully uploaded."""
    conn = get_db()
    conn.executemany(
        "UPDATE sensor_readings SET synced = 1 WHERE id = ?",
        [(rid,) for rid in row_ids]
    )
    conn.commit()

Quick Reference

Do these from the start. Enable WAL mode and synchronous=NORMAL immediately after opening each connection. Use parameterised queries — never string concatenation. Add a timestamp index and a synced column if you are buffering for later upload. Set a retention policy before your first deployment.

Avoid these. Do not commit per row in a high-frequency logging loop — batch your inserts. Do not share a connection across threads. Do not call VACUUM on every cleanup. Do not use the default journal mode if you have concurrent readers and writers.

Use when you need it. executemany for bulk inserts. row_factory = sqlite3.Row for readable query results. VACUUM once a day or week to reclaim space. PRAGMA integrity_check after a power loss if you want to verify the database is undamaged.

If you found this useful, drop a comment or a ❤️ — and let me know what you'd like to see covered next.

DuckDB vs SQLite: Which one is better?

Supun Sriyananda — Tue, 09 Jun 2026 09:18:52 +0000

I spend most of my time somewhere between microcontrollers and dashboards. One week I'm squeezing firmware onto a device with barely any memory to spare, the next I'm staring at a few million sensor readings trying to work out why a gateway in the field keeps misbehaving. So when people line up "DuckDB vs SQLite" as a fight to the death, I always want to gently jump in.

They're both small. They both run inside your own program instead of off on some server. They both let you work with a single file using SQL. On paper that makes them sound like rivals. But the more I've used them — across embedded work, edge devices, and plain old data crunching — the more they feel like two neighbours who happen to do completely different jobs.

Let me walk through what I mean, starting from the very beginning.

First, what "embedded" actually means here

When people hear the word "database," they often picture something big and separate: a program running on its own server somewhere that your app has to connect to over the network, with a username and password, before it can read or write anything. PostgreSQL and MySQL work like that. There's nothing wrong with it — it's how most large web apps run — but it's a lot of moving parts.

SQLite and DuckDB throw all of that out. There's no separate program to start. Nothing to log into. No server quietly running in the background. The whole database is just an ordinary file sitting on your disk, and the database engine is a small library your code loads in directly. You point it at the file, you run your queries, and that's the whole setup. Nothing else to install or babysit.

That shared simplicity is the lovely part, and it's exactly why you find these two in places a big server-based database could never go — phones, web browsers, tiny sensors, a quick script on your laptop. But the moment you look at how each one stores your data and works through it, they head off in opposite directions. That difference is the whole story, so let's go there next.

SQLite: the one that's already everywhere

SQLite stores data in rows. Picture a spreadsheet where each row is one complete record, and all the values for that record sit together: the id, the name, the temperature reading, all in one place. That layout is perfect when you're constantly poking at individual records — add this new reading, update that setting, grab the latest entry for device 42. You want the whole record at once, and SQLite hands it to you fast.

A few things I love about it:

It's tiny. The whole engine is around a megabyte. On a small device, that matters enormously.
It's everywhere, and I mean everywhere. There are tens of billions of SQLite files in active use — it's running on basically every phone, and your web browser uses it right now to store your history and settings. It has earned the right to be boring.
It's astonishingly reliable. It's one of the most thoroughly tested pieces of software on the planet. The US Library of Congress even recommends SQLite as a format for preserving digital files long-term, because they trust it'll still open decades from now. When you're shipping a device that has to run untouched in a cabinet for years, that track record buys you a lot of peace of mind.

In my embedded work, SQLite is the default for anything that holds state — the current situation the device needs to remember. Its settings. A queue of readings waiting to be uploaded. A small log of recent events. It handles a steady stream of small writes gracefully and it doesn't ask for resources the hardware doesn't have.

Where it starts to struggle is heavy number-crunching across huge piles of data. Ask SQLite to add up and average ten million rows and it'll get there — but slowly, because it reads through the data row by row, dragging along every column even when your question only touches one of them. That's not a flaw. It simply wasn't built for that job.

DuckDB: the one that makes big analysis feel easy

DuckDB flips the storage layout around. Instead of keeping each row together, it keeps each column together — all the temperatures in one place, all the device names in another. So when you ask "what's the average temperature across ten million readings," it reads just the temperature column and skips everything else. On top of that it processes data in big batches rather than one row at a time, which modern processors are very good at. The result is that the heavy questions that make SQLite sweat come back from DuckDB before you've finished a sip of coffee.

There's another part that genuinely changed how I work, and it has nothing to do with speed. DuckDB will read your data files directly, right where they sit:

SELECT device_id, avg(temperature)
FROM 'readings/*.parquet'
GROUP BY device_id;

No loading step. No importing anything into a table first. You point it at a folder of files — CSV, JSON, or Parquet (a compact file format that's common for this kind of data) — and it just reads them. For someone who regularly gets handed a pile of sensor dumps, that's the difference between "give me an afternoon to set up a pipeline" and "give me thirty seconds."

It also handles data bigger than your computer's memory by spilling the overflow to disk, so a dataset that would crash a normal in-memory tool just... works.

It's newer than SQLite and the engine is a bit chunkier, but it's still the same idea at heart: a file and a small library, nothing to run separately.

Are these actually used in the real world?

Both, heavily — this isn't a case of betting on something obscure.

SQLite is the most widely deployed database engine that exists, full stop. The tens of billions of copies in daily use make it more common than every other database combined. It's not going anywhere.

DuckDB is younger but its adoption has shot up. It's pulling in around 37 million downloads a month on Python's package index, it's MIT licensed and free, and it has real commercial backing behind it (a company called MotherDuck builds a cloud service on top while keeping the core engine open and free), which answers the usual worry about whether an open-source tool will still be maintained in five years. It also fits neatly with where the industry is heading: open file formats, modern multi-core processors, and even AI coding assistants, which tend to be good at writing SQL and so reach for DuckDB naturally.

And SQLite isn't standing still either. Newer spin-offs like Turso/libSQL are adding things like replication and edge support on top of the classic engine. Both of these tools are safe bets.

So when does each one actually win?

Here's how it shakes out across the three places I work.

On small, constrained hardware, SQLite, almost every time. DuckDB's appetite for memory and processing power during a big query is more than a tiny chip wants to give. On a larger edge device running Linux it's a different story, but down at the small end, SQLite's tiny size and long history are hard to argue with.

At the edge — meaning the gateway boxes that sit between your sensors and the cloud — they actually team up. A pattern I keep coming back to: let SQLite handle the incoming readings on the device, quietly buffering them as they arrive, then let DuckDB do the local number-crunching before anything gets sent upstream. You ship neat summaries instead of the raw firehose, which is kinder to both your bandwidth bill and your cloud costs. They're not competing here. They're a relay team.

For sitting down and analysing data, DuckDB is the one I reach for. Being able to throw SQL at a folder of files without setting up any heavy machinery is exactly the kind of low-fuss tool the job usually calls for. It has quietly become a go-to for local analysis, and for good reason.

The one line I keep in my head

If I had to shrink all of this down to a fridge magnet:

SQLite is for managing data while it's being created and changed. DuckDB is for analysing it once it's all piled up.

Same small, no-server spirit — opposite ends of the data's life. SQLite looks after the data as it's coming in and changing. DuckDB shows up later to make sense of the whole pile. Once that clicked for me, the "versus" framing kind of fell apart.

So if you're choosing between them, the honest answer is often both, for different things. Work out whether the job in front of you is about handling data as it changes or making sense of a big pile of it, and the choice mostly makes itself.

If you've wired these two together in your own projects, I'd love to hear how you split the work. I'm always tinkering with my own setup, and the edge folks always seem to have the most interesting war stories.

Deploying Production Systems on Raspberry Pi: Lessons from the Field

Supun Sriyananda — Sun, 07 Jun 2026 05:38:31 +0000

Deploying Production Systems on Raspberry Pi: Lessons from the Field

These are the things I wish I had known before deploying Pis in production.

SD Cards Will Kill You

The first production Pi I deployed used a generic microSD card. It failed after four months. The second one used a "name brand" card. It failed after six months. The pattern remained always the same: the filesystem corrupts during a power loss, the Pi boots into read-only mode, and whatever the system was supposed to be doing silently stops working.

SD card corruption under power loss is not a bug you can fix in software. It is a fundamental characteristic of flash storage that was designed for cameras, not servers. The cells wear out, write operations are not atomic, and a sudden power cut mid-write leaves the filesystem in a state that fsck (File System Consistency Check) sometimes cannot recover.

If you choose SD cards: Switch to a Pi-rated industrial SD card (SanDisk MAX Endurance, Samsung Pro Endurance) or eliminate the SD card entirely by booting from a USB SSD. USB boot on Pi 4 and Pi 5 is stable and the endurance difference is enormous — a decent SSD handles orders of magnitude more write cycles than any SD card.

For systems that must use SD, mount the filesystem read-only and put all writable state on a tmpfs or a separate partition with journaling.

tmpfs is a special type of temporary file storage facility in Linux and Unix-like systems that stores files directly in volatile memory (RAM) instead of on a persistent drive like an SD card or SSD.

When you mount a folder as tmpfs, any files written to that folder behave like regular files, but they consume RAM and exist purely in memory.

# /etc/fstab — mount root read-only, put logs and data elsewhere
/dev/mmcblk0p2  /        ext4  ro,defaults  0  1
tmpfs           /tmp     tmpfs defaults      0  0
tmpfs           /var/log tmpfs defaults      0  0

# Separate partition for application data with journaling
/dev/mmcblk0p3  /var/lib/myapp  ext4  defaults,data=journal  0  2

For a warehouse sensor deployment, the application writes data to SQLite on a separate ext4 partition with journaling enabled. If the Pi loses power mid-write, fsck can recover the journal. The root partition is read-only and survives power loss cleanly every time.

Thermal Throttling Is Silent and Intermittent

The thing is Pi will not tell you that it is throttling. It will not log a warning. That means, your video stream will just start dropping frames, your serial latency will increase, and your MQTT reconnects will take longer. And as you can see, all these symptoms look like software bugs. But if you check:

vcgencmd get_throttled

vcgencmd get_throttled is a command line tool unique to the Raspberry Pi that checks whether the computer has lowered its CPU speed.

0x0 means everything is fine. Anything else means the Pi is throttling now or has throttled since last boot. The common values:

0x50005 — The danger zone. You are currently under-volted and throttled right now, and it has happened before.
0x50000 — This means your system is physically running fine right now, but under-voltage (0x10000) and throttling (0x40000) have occurred in the past. Your current power supply is dropping voltage under load, making your SD card highly vulnerable to corruption.
0x4 — soft temperature limit active

I was building a telepresence robot, encoding 720p30 H.264 while running the aiohttp server and serial communication was pushing the Pi 4 to 80°C without a heatsink. The encoder started dropping frames randomly. Adding a heatsink brought idle temperature to 45°C and load temperature to 62°C. Managed to get the throttling under control.

On a Pi 5, the situation is better but not solved. The Pi 5 has an active cooler as an official accessory and it is worth using in any deployment where the Pi is in an enclosure. But, enclosures trap heat. A Pi in a plastic project box with no airflow will throttle faster than a bare board.

Also it is a best practice to add temperature monitoring to your health check endpoint so you find out before users do:

import subprocess

def get_pi_health():
    temp = subprocess.run(
        ['vcgencmd', 'measure_temp'],
        capture_output=True, text=True
    ).stdout.strip()  # "temp=58.0'C"

    throttled = subprocess.run(
        ['vcgencmd', 'get_throttled'],
        capture_output=True, text=True
    ).stdout.strip()  # "throttled=0x0"

    return {
        'temperature': temp,
        'throttled': throttled,
        'throttled_ok': throttled == 'throttled=0x0'
    }

Power Supply Quality Matters More Than Rated Current

A power supply rated for 3A is not always a 3A power supply. Cheap USB-C supplies have poor voltage regulation. So, under a load they sag below the 5.1V the Pi needs and trigger under-voltage throttling. The symptom is identical to thermal throttling: random slowdowns, occasional reboots, SD card corruption on shutdown.

The official Raspberry Pi power supply is not a premium product — it is a specification-compliant one. Use it, or use a bench power supply for development and a known-good supply for deployment. The Pi 5 draws up to 5A under full load; a 3A supply will cause under-voltage events when the CPU is fully loaded.

For any deployment running off mains power, a UPS hat (Geekworm UPS hat, Waveshare UPS hat) is worth the £20. The Pi gets notified of incoming power loss and can initiate a clean shutdown before the battery dies, which eliminates the entire class of "power cut during SD write" corruption events.

Network time synchronization errors can cause unexpected system failures

A Pi that has been offline for an extended period will have a wrong system clock when it boots — sometimes wrong by days if the RTC battery is dead or there is no RTC at all. Applications that timestamp log entries, certificate validity checks, and SQLite timestamp comparisons all behave unexpectedly when the system time is wrong.

A specific failure case I encountered: the MQTT client was writing timestamps using datetime.now().isoformat(). After a boot without internet, the system clock was set to 2023-01-01 (the default). All queued messages got timestamps in 2023. When the clock corrected to 2024 via NTP after network connection, the retention policy deleted those messages as being "older than 7 days" — because relative to the current time they appeared to be a year old.

Fix 1: Use a hardware RTC. The DS3231 costs about £3 and keeps accurate time across power cycles without network. Enable it with a device tree overlay:

# /boot/firmware/config.txt
dtoverlay=i2c-rtc,ds3231

Fix 2: For timestamps that survive offline periods, use monotonic time for intervals and NTP-synced time only for absolute timestamps. Do not mix them.

Fix 3: Configure chrony or systemd-timesyncd to be aggressive about syncing on boot and to accept large time jumps:

# /etc/chrony.conf
makestep 1 -1   # Accept any step size, any number of times

Watchdog Timers Are Not Optional

If your Pi is in a location where you cannot physically reach it — mounted on a robot, installed in a warehouse, bolted inside a wall panel — a software crash or infinite loop that freezes the application is effectively a permanent failure until someone intervenes.

The Linux kernel watchdog kicks the hardware watchdog timer while the kernel is running. If the kernel hangs, the watchdog expires and forces a reboot. But it does not know whether your application is running correctly. For that, you need an application-level watchdog.

systemd's built-in watchdog support requires almost no code:

# In your main loop, notify systemd you're still alive
import os
import time

def notify_watchdog():
    """Tell systemd the application is healthy."""
    os.system("systemd-notify WATCHDOG=1")

# Call this from your main loop — if you stop calling it,
# systemd will restart the service
while True:
    do_work()
    notify_watchdog()
    time.sleep(5)

# /etc/systemd/system/myapp.service
[Service]
WatchdogSec=30        # Restart if no heartbeat for 30 seconds
Restart=always
RestartSec=5
StartLimitIntervalSec=120
StartLimitBurst=5

For applications where notify_watchdog() cannot be called from the main loop (async applications, multi-threaded servers), run it from a background thread that monitors the health of the main thread.

Remote Access Must Work Before Anything Breaks

The time to set up remote access is before you deploy, not after. I use Tailscale for most Pi deployments because it takes five minutes to configure, works through NAT, does not require port forwarding, and uses WireGuard under the hood. Once it is running, you have a reliable backdoor to your hardware. However, if you need more control, complete data sovereignty, and zero third-party dependencies, use vanilla WireGuard instead. While WireGuard requires you to manually configure routing rules and host a central server with an open port for NAT traversal, it gives you total ownership over your network topology without device or account limitations:

# Install
curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up

# Enable SSH in your tailnet policy and you can reach the Pi from anywhere
ssh user@cyrobot.turkey-trench.ts.net

Tailscale also provides valid TLS certificates for your device's hostname in the tailnet — which is how the WebRTC server serves HTTPS without a domain name or public CA.

Set up mosh alongside SSH for unreliable connections. Regular SSH sessions die when the network hiccups. Mosh sessions survive.

Logs Fill Up the Filesystem

/var/log on a production Pi will fill up over months of continuous operation. When it does, your application cannot write logs, SQLite cannot open its WAL file, and things fail in confusing ways that do not obviously point to "disk full."

Set up log rotation from day one:

# /etc/logrotate.d/myapp
/var/log/myapp/*.log {
    daily
    rotate 7
    compress
    missingok
    notifempty
    size 10M
}

And add disk usage to your health check:

import shutil

def check_disk():
    usage = shutil.disk_usage('/')
    free_percent = (usage.free / usage.total) * 100
    return {
        'free_gb': round(usage.free / 1e9, 2),
        'used_percent': round(100 - free_percent, 1),
        'warning': free_percent < 15
    }

The Summary

Before deploying any Pi to a location you cannot easily reach:

Boot storage is an industrial SD card or USB SSD. The root filesystem is read-only or has journaling on writable partitions. A hardware RTC is installed. A heatsink or active cooler is fitted. A quality power supply is used and a UPS hat is fitted if on mains power. Tailscale is installed and tested from outside the local network. systemd service has Restart=always, WatchdogSec, and StartLimitBurst set. Log rotation is configured. A health endpoint exposes temperature, throttle status, and disk usage. You have confirmed you can SSH in and restart the service from the office before going to the deployment site.

Written by Supun Akalanka | Category: Lessons Learned | Tags: Raspberry Pi, Production, Reliability, Embedded Linux, Hardware

Creating Robust systemd Services for Embedded Applications

Supun Sriyananda — Wed, 03 Jun 2026 09:51:04 +0000

There is a moment every embedded Linux developer hits eventually. You have spent days building something that works beautifully — a sensor pipeline, a streaming server, an MQTT client — and then you reboot the device and everything is silent. Nothing started. You SSH in, manually run your script, and it all comes back to life. The hardware is fine. Your code is fine. You just have no way of automatically running it.

That is the gap systemd fills. It is the init system on virtually every modern Linux distribution, and on embedded Linux systems like the Raspberry Pi it is what decides what runs at boot, what gets restarted if it crashes, and where all the logs go. Once you understand how to write a service file, your applications stop being fragile scripts you need to babysit and start being first-class system services that survive reboots, network drops, and unexpected crashes.

This tutorial builds up from the simplest possible service file to a production-ready configuration, explaining every line along the way. By the end you will have a service running your own Python application, logging to the system journal, and automatically restarting itself after failures.

See Complete Tutorial in Github: Systemd Services Tutorial

What systemd Actually Does

Before writing any configuration, it helps to understand what problem systemd is solving, because the design of service files makes much more sense once you see the underlying model.

When your Raspberry Pi boots, the Linux kernel starts and immediately hands control to process ID 1 — the very first user-space process. On modern systems, that process is systemd. Everything that happens next — mounting filesystems, bringing up the network, starting your application — is orchestrated by systemd. It reads configuration files called unit files that describe what should be started, when, in what order, and what to do if something goes wrong.

A service file is just one type of unit file (there are also unit files for timers, sockets, mount points, and more, but services are what you will use most). When you tell systemd about your application through a service file, you are essentially saying: "here is my program, here is when I want it to run, and here is how I want you to manage it." systemd takes it from there — starting it at boot, watching it, restarting it if it dies, and capturing everything it prints to stdout and stderr into a structured log called the journal.

The central tool for interacting with systemd is systemctl. You use it to start, stop, enable, disable, and inspect services. The companion tool journalctl gives you access to the journal — the logs that systemd collects from every service it manages.

A Minimal Service File

Let us start with the simplest possible service to understand the structure, then build from there. Suppose you have a Python script at /home/pi/mqtt_client/production_client.py that you want to run automatically at boot.

Service files live in /etc/systemd/system/. Create a new one:

sudo nano /etc/systemd/system/mqtt-client.service

Here is the minimal version:

[Unit]
Description=Production MQTT Edge Client

[Service]
ExecStart=/usr/bin/python3 /home/pi/mqtt_client/production_client.py

[Install]
WantedBy=multi-user.target

Even this tiny file has three sections and each one has a specific job. The [Unit] section contains metadata and dependency declarations — the Description is just a human-readable label that appears in logs and status output. The [Service] section is where the actual execution configuration lives — right now we only have ExecStart, which is the command that launches your program. The [Install] section controls how the service integrates into the boot process — WantedBy=multi-user.target means "start this service when the system reaches the normal multi-user state", which is essentially "start this at boot when the system is ready for normal operation."

To activate it, you need to do two things: reload systemd so it picks up the new file, and enable the service so it starts at boot:

# Tell systemd to re-read all unit files
sudo systemctl daemon-reload

# Enable it to start at boot
sudo systemctl enable mqtt-client.service

# Start it right now without waiting for a reboot
sudo systemctl start mqtt-client.service

Check whether it is running:

sudo systemctl status mqtt-client.service

You will see output that tells you the service state active (running). This is what you want, the process ID, when it started, and the last few lines of log output. If it failed, the status output will usually tell you exactly why — an incorrect path, a Python import error, whatever the problem is.

The difference between enable and start trips people up at first, so it is worth being explicit: enable creates a symlink that tells systemd to start the service at boot — it does not start it right now. start starts it immediately — it does not persist across reboots. In practice you almost always want both.

Making it Actually Robust: Restart Policies

The minimal service above will start your application at boot, but if the application crashes — which happens, especially in edge environments with unreliable hardware or network — systemd will not do anything. It will just leave the service in a failed state. For an embedded device running unattended in the field, that is not acceptable.

The Restart directive in the [Service] section tells systemd what to do when your application exits. Here is what the options mean in practice. no means never restart (the default — not what you want for production). on-failure means restart if the process exits with a non-zero exit code or is killed by a signal, but not if it exits cleanly with code 0. always means restart no matter what — even if the process exits with code 0. on-abnormal means restart on crash or signal, but not on clean exit or timeout.

For most embedded applications, on-failure is the right choice. It means "if something goes wrong and the program dies unexpectedly, bring it back", but it also means "if I deliberately stop the service with systemctl stop, do not restart it."

There is one more important setting to pair with Restart: RestartSec, which sets how long to wait before restarting. Without it, systemd restarts immediately, which can cause problems if your service is crashing due to a dependency that is not ready yet (like the network). A short delay — even two or three seconds — gives things time to settle.

Here is the updated service with restart policy:

[Unit]
Description=Production MQTT Edge Client

[Service]
ExecStart=/usr/bin/python3 /home/pi/mqtt_client/production_client.py
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

Declaring Dependencies

This is where systemd really starts to earn its keep. Most embedded applications do not run in isolation — they depend on the network being up, a filesystem being mounted, or another service being ready. systemd lets you express these dependencies explicitly so that your service starts at the right time and in the right order.

The two most commonly confused directives here are After and Requires. Think of them as answering different questions. After=network.target answers the question "when should I start?" — it tells systemd to not even attempt to launch this service until the network target has been reached. Requires=network.target answers the question "what must exist for me to function?" — it tells systemd that if the network goes away, this service should be stopped too.

You can use them together, and for network-dependent applications you almost always should. There is also Wants, which is a softer version of Requires — it expresses preference rather than hard dependency. If what Wants declares is not available, the service will still start rather than failing.

For an MQTT client that needs the network to do anything useful:

[Unit]
Description=Production MQTT Edge Client
After=network.target
Wants=network.target

[Service]
ExecStart=/usr/bin/python3 /home/pi/mqtt_client/production_client.py
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

Wants rather than Requires here is deliberate. The MQTT client already handles network loss internally — it has exponential backoff and an offline queue for exactly this scenario. So we do not want systemd to kill the service if the network drops; we just want to make sure we do not start before the network stack is even initialised.

For cases where your application would be completely broken without a dependency — a database-backed service where the database is also managed by systemd, for example — use Requires. The distinction matters in practice.

Working Directory and Environment

Two things that bite almost every developer on their first real systemd deployment are working directory and environment variables. When you run a script manually from your terminal, you have a working directory (usually your home folder) and a full set of environment variables inherited from your shell. systemd does not give you either of those by default.

This matters because Python scripts often use relative paths like ./config.json or ./logs/, and those paths will fail when the working directory is not what you expect. Environment variables like PATH, HOME, and any custom variables your application reads from the environment will also be missing or wrong.

WorkingDirectory solves the path problem, and Environment or EnvironmentFile solves the variable problem:

[Unit]
Description=Production MQTT Edge Client
After=network.target
Wants=network.target

[Service]
# Set the working directory so relative paths work correctly
WorkingDirectory=/home/pi/mqtt_client

# Pass environment variables directly
Environment=MQTT_BROKER_HOST=192.168.1.100
Environment=MQTT_CLIENT_ID=warehouse_sensor_01

# Or load them from a file (better for secrets)
# EnvironmentFile=/etc/mqtt-client/config.env

ExecStart=/usr/bin/python3 /home/pi/mqtt_client/production_client.py
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

The EnvironmentFile approach is worth knowing about even if you do not use it immediately. It lets you store configuration in a separate file — including secrets like passwords — outside the service file itself. A file like /etc/mqtt-client/config.env contains simple KEY=value lines and is loaded by systemd before starting the service. This keeps credentials out of version control and makes configuration changes possible without touching the service file.

Logging

One of systemd's genuine gifts to embedded developers is centralised logging. Everything your application prints to stdout and stderr is automatically captured by the journal — no log file configuration required, no rotation to set up. The journal is structured, persistent across reboots (on most systems), and queryable in powerful ways.

The basic command to read logs from your service is:

# Show all logs from your service
journalctl -u mqtt-client.service

# Follow logs in real-time (like tail -f)
journalctl -u mqtt-client.service -f

# Show only logs since the last boot
journalctl -u mqtt-client.service -b

# Show logs from the last hour
journalctl -u mqtt-client.service --since "1 hour ago"

For embedded devices with limited storage, it is worth explicitly configuring how much disk space the journal is allowed to use. You do this not in the service file but in the journal configuration:

sudo nano /etc/systemd/journald.conf

Add or modify these lines:

[Journal]
# Maximum journal size on disk
SystemMaxUse=50M

# Maximum size of a single journal file
SystemMaxFileSize=10M

Then restart the journal daemon:

sudo systemctl restart systemd-journald

For embedded deployments where the device has a small SD card or eMMC, keeping the journal small is important. Fifty megabytes is a reasonable limit for most edge devices.

A Production-Ready Example

Putting all of this together, here is what the service file looks like for a real deployment — the kind of configuration you would use for a long-running embedded application that needs to be reliable in the field. This is modelled after exactly how I deploy the MQTT edge client and the WebRTC streaming service on the telepresence robot:

[Unit]
Description=Production MQTT Edge Client with Offline Resilience
# Human-readable detail shown in status output and logs
Documentation=https://github.com/ranaweerasupun/mqtt-production-client

# Start after the network is available, but do not require it —
# the application handles network loss internally with its own
# backoff and offline queue
After=network.target
Wants=network.target

# If the service restarts more than 5 times in 60 seconds, give up
# This prevents an infinite crash-restart-crash loop from overwhelming the system
# (these belong in [Unit] on systemd v230+)
StartLimitIntervalSec=60
StartLimitBurst=5

[Service]
# Run as a specific user rather than root — better security practice
User=pi
Group=pi

# Set working directory so relative paths in the application work
WorkingDirectory=/home/pi/mqtt_client

# Load configuration from a separate file
# This keeps secrets out of the service file
EnvironmentFile=/etc/mqtt-client/config.env

# The command that starts the application
ExecStart=/usr/bin/python3 /home/pi/mqtt_client/production_client.py

# What to do when the process exits unexpectedly
# on-failure: restart if the process crashes or exits non-zero
# but NOT if you run 'systemctl stop mqtt-client'
Restart=on-failure

# Wait 5 seconds before restarting
# This prevents rapid restart loops if something is fundamentally broken
RestartSec=5

# Give the service up to 30 seconds to stop gracefully before killing it
TimeoutStopSec=30

# Capture stdout and stderr to the journal
StandardOutput=journal
StandardError=journal

# Tag journal entries with this identifier for easy filtering
SyslogIdentifier=mqtt-client

[Install]
WantedBy=multi-user.target

The StartLimitIntervalSec and StartLimitBurst combination is worth understanding because it solves a real problem. Imagine your MQTT client has a bug that makes it crash immediately on startup — perhaps a malformed config file or a missing dependency. Without these limits, systemd would restart it immediately, it would crash again, systemd would restart it again, and this would loop forever, consuming CPU and filling your journal with crash logs. With StartLimitBurst=5 and StartLimitIntervalSec=60, systemd will make five restart attempts within a 60-second window, and if all five fail, it marks the service as failed and stops trying. At that point systemctl status will clearly tell you the service has hit its restart limit, which prompts you to actually investigate the root cause.

The User=pi directive is also important for production embedded deployments. Running services as root is a security risk — if your MQTT client has a vulnerability, an attacker who exploits it gets root access. Running as a non-privileged user limits the damage. The tradeoff is that you need to make sure that user has permission to access the files and ports your application needs.

Useful Commands to Know

Once your service is running, a handful of commands will cover most of what you need day-to-day:

# Check current status, recent logs, and whether it is enabled at boot
sudo systemctl status mqtt-client.service

# Start the service right now
sudo systemctl start mqtt-client.service

# Stop the service (will not restart automatically due to Restart=on-failure)
sudo systemctl stop mqtt-client.service

# Restart the service (useful after changing your application code)
sudo systemctl restart mqtt-client.service

# Reload the service file after you edit it (then restart to apply changes)
sudo systemctl daemon-reload
sudo systemctl restart mqtt-client.service

# Disable the service from starting at boot
sudo systemctl disable mqtt-client.service

# View full log history for this service
journalctl -u mqtt-client.service

# View logs in real-time
journalctl -u mqtt-client.service -f

One workflow note: whenever you edit the service file, you must run daemon-reload before the changes take effect. systemd caches unit file contents and will not notice your edits otherwise. Forgetting this step is a common source of confusion when your changes do not seem to be working.

What systemd Gives You for Free

It is worth pausing to appreciate what you get from writing a proper service file, because the alternative — ad hoc startup scripts in /etc/rc.local or cron @reboot jobs — gives you almost none of it.

With a systemd service you get automatic restart on crash, ordered startup relative to other system components, centralised and queryable logging, a clean mechanism to start and stop the application during development, restart rate limiting to prevent crash loops, graceful shutdown handling, and the ability to run as a non-root user easily. All of that from a text file that is, at its core, less than twenty lines.

For an embedded device deployed in a factory, a warehouse, or anywhere else it needs to run unattended for months at a time, that reliability infrastructure is not optional. systemd gives it to you essentially for free, as long as you take the time to describe your service correctly.

Finishing What I Started — From a TODO List to a Published PyPI Package

Supun Sriyananda — Thu, 28 May 2026 13:03:03 +0000

This is a submission for the GitHub Finish-Up-A-Thon Challenge

What I Built

I built robmqtt — a resilient MQTT client for edge IoT devices, now published on PyPI.

It came out of a problem that cost me real time. I deploy sensor systems on hardware that lives in the field: Raspberry Pi units on 4G cellular, embedded gateways, battery monitoring systems. On those networks, connectivity is never stable. And the standard MQTT client, paho-mqtt, silently drops messages when the broker is unreachable — no error, no warning, no log entry unless you write one yourself.

I lost data for a long time before I understood why. And finding the cause was brutal. There are no proper logs unless you explicitly add them, so I spent hours — days — chasing a problem that left no trace. I went through forums, blogs, chat AIs, everywhere, looking for how other people had solved it. What I found was the same thing every time: people asking for a production-resilient MQTT client, and no real answer. Just scattered advice and code snippets that handled one piece and ignored the rest.

So I built it myself:

Offline queue — when the broker is unreachable, messages are written to SQLite instead of being dropped. They survive process restarts and power cycles.
Inflight tracking — messages sent but not yet acknowledged are tracked separately and re-sent on reconnect, closing a gap that even QoS 1 leaves open.
Priority eviction — when the queue fills, low-priority telemetry is evicted before critical alerts.
Exponential backoff — a fleet of devices reconnecting after an outage won't all hammer the broker at once.
TLS support — for brokers that require encrypted connections.

And then everything a library needs to actually be used in production, not just by me:

TLS and mutual TLS plus username/password auth, for secured brokers.
Bidirectional messaging — subscribe/unsubscribe with full MQTT wildcard support, and subscriptions that survive reconnects.
Observability — a built-in HTTP health check endpoint (/health returning healthy / degraded / unhealthy) with Docker HEALTHCHECK and Kubernetes liveness-probe examples.
Structured logging throughout — so the next person doesn't lose days to a problem with no trace, the way I did.
73 tests across the storage, tracking, topic-matching, and client layers.

pip install robmqtt

This matters to me because it's not a toy. It's running on real systems I maintain — battery management monitoring and robotics telemetry — where a gap in the data record has actual consequences.

Demo

PyPI: https://pypi.org/project/robmqtt/
GitHub: https://github.com/ranaweerasupun/resilient-edge-mqtt-client

The most satisfying way to see it work is to watch the offline queue survive a broker outage. The repo includes a simulation script for exactly this:

# Terminal 1 — run a simulated device publishing every 5 seconds
python test_13.py

# Terminal 2 — kill the broker mid-run
sudo systemctl stop mosquitto

The device keeps publishing. Messages start queuing to SQLite instead of erroring. Then:

# Bring the broker back
sudo systemctl start mosquitto

The queue drains automatically, in priority order. Every message that piled up during the outage is delivered. Nothing is lost.

Basic usage looks like this — the application code never has to know whether the broker is reachable:

from robmqtt import ProductionMQTTClient
import json, time

client = ProductionMQTTClient(
    client_id="field_device_001",
    broker_host="mqtt.yourdomain.com",
    broker_port=1883,
    max_queue_size=5000,
    db_path="./device.db",
)
client.connect()
client.start()

while True:
    client.publish(
        topic="sensors/temperature",
        payload=json.dumps(read_sensor()),
        qos=1,
        priority=5,
    )
    time.sleep(30)

The Comeback Story

Here's the honest before-and-after.

Before. Once I'd finally figured out the code that solved my problem, I used it — and it went into my GitHub repo, where it sat. It worked. It solved the thing that had cost me days. But no one knew it existed.

And that was the part that nagged at me. I'd keep seeing people online asking for exactly what I'd built — something resilient enough for production — and getting the same non-answers I'd gotten: random advice, half-solutions, snippets. I had the answer sitting in a corner of my repo, and a thought stuck in the corner of my head: it's right here. I need to show this off.

The things I knew I should do but hadn't:

Package it for PyPI so people could pip install it instead of cloning the repo
Add TLS/mTLS support for secure broker connections
Expose a metrics endpoint for observability

It was the classic unfinished side project. Functional, but not shipped. Not something anyone other than me could actually use. The plan was always "publish it once it's polished" — and polishing it was the part that kept not happening.

The hard part. I have a full-time job. Finishing this wasn't something I could do in work hours. So it happened on weekends, late at night, and in the gaps — I drew flowcharts and planned the package structure on my commute, in my head, on paper. Progress came in small pieces, squeezed around everything else. There were plenty of stretches where it would have been easier to just leave it in the repo and move on.

After. I pushed it over the line:

Published to PyPI as robmqtt v1.0.0. This was more work than I expected — restructuring the code into a proper installable package, writing the pyproject.toml, sorting out the module layout, testing the install in a clean environment.
Added TLS support. The client now takes use_tls, ca_certs, certfile, keyfile, and broker auth parameters, so it works with secured brokers, not just local plaintext ones.
Observability Built an HTTP health check endpoint with Docker and Kubernetes probe examples — so the library reports its own health instead of failing silently.
Structured logging so the next person doesn't suffer days to a problem with no trace, the way I did!
Data Integrity Fixed three bugs I'm glad I caught before anyone relied on it: binary payloads being corrupted by string conversion, a resend-tracking gap that could silently lose messages on a second disconnect, and a race condition in publish() that could drop a message between the connection check and the send.
Started building a real project on top of it. To put the library through its paces in a full system, I'm building an IoT fleet analytics platform around it — simulated devices publishing through robmqtt, an MQTT-to-InfluxDB bridge, Grafana dashboards, and statistical anomaly detection. It's still in progress, but it's already doing what mattered most: proving the library holds up as the foundation of a real pipeline, not just in isolated tests.

When it was finally published and I ran the test scripts and watched the offline queue fill and drain exactly the way it was supposed to — messages held through an outage, then delivered, nothing lost — there was nothing more rewarding. The thing that had cost me days of frustration was now one pip install away for anyone who hits the same wall I did.

My Experience with GitHub Copilot

I'll be honest about my workflow, because the reason I relied on Copilot is specific.

I use a few AI tools when I build things. The general-purpose chat assistants are useful for sketching out an approach or talking through a design — but when it came to actual code, they kept giving me snippets that didn't quite work. That's not surprising: they can't see my project. They don't know my file structure, my variable names, the exact version of a library I'm using, or how the piece they're suggesting fits the rest of the code. So I'd paste in a snippet, hit an error, paste the error back, get a revised snippet, hit another error. Going back and forth with a chat window that can't see my codebase got slow and frustrating.

Copilot was different because it lives in my editor and can see everything — all my open files, the actual code around the cursor, the real context. That's why it became the tool I leaned on.

Two ways I used it:

Inline completion, always on. As I typed, Copilot autocompleted the repetitive, structural parts — the device profile dictionaries, the JSON payload construction, the boilerplate around publish() calls with their QoS and priority arguments. Because it could see the patterns already in my file, its suggestions actually fit, instead of being generically plausible code I'd have to rewrite.

Copilot Chat for debugging in context. This is where it earned its place. When something broke, I'd ask it directly — "Why am I getting this error, and how can I fix it?" or "Can you check this code snippet?" — and because it could read my actual files, the answers were grounded in my real code, not a guess about what my code might look like. That's the difference that made me stop pasting snippets into external chat windows. The tool that can see your codebase gives you answers that apply to your codebase.

That freed me to spend my real thinking time on the parts that mattered: the sine-wave drift model for realistic sensor data, the priority eviction logic, and the inflight-tracking design that closes the QoS 1 gap. The hard design decisions were mine. Copilot removed the friction around them — the typing, the boilerplate, and the slow debugging loop — so I could stay focused on the actual engineering.

Built by Supun Sriyananda — R&D Engineer working on embedded and IoT systems. robmqtt is open source on GitHub and PyPI.

How to Generate Realistic IoT Sensor Data for Testing Your MQTT Pipeline

Supun Sriyananda — Thu, 28 May 2026 11:02:48 +0000

How to Generate Realistic IoT Sensor Data for Testing Your MQTT Pipeline

This is part 2 of a series on building robmqtt. Part 1 covered why paho-mqtt silently drops messages and the library I built to fix it. This part is about testing — how to exercise an MQTT pipeline without deploying physical hardware.

In the last post I wrote about robmqtt, a resilient MQTT client for edge devices. Once I had it working, I hit the obvious next problem: how do I test it properly without setting up a rack of Raspberry Pis?

I needed data flowing through the pipeline. Lots of it. From many devices. Behaving differently. And ideally surviving a broker outage so I could watch the offline queue do its job.

So I wrote a device simulator. And in writing it, I learned that generating realistic fake sensor data is harder than it looks — and that most people get it wrong in the same way.

The trap: random data isn't realistic data

The first instinct when simulating a sensor is to reach for random.uniform():

cpu = random.uniform(0, 100)   # don't do this

This produces data that looks nothing like a real sensor. Real sensor readings don't jump randomly across the whole range every second. A CPU sitting at 8% doesn't suddenly read 94% then drop to 3%. Temperature drifts slowly. Signal strength wobbles around a baseline. There's continuity from one reading to the next.

If you test your pipeline with pure random noise, your charts look like static, your anomaly detection has nothing meaningful to detect, and your dashboards are useless for spotting whether anything actually works.

I wanted data that looked like it came from a real device.

Realistic drift with a sine wave

The trick I landed on was a slow sine wave with per-device random phase, plus a little Gaussian noise on top:

def _cpu(self) -> float:
    p = self.profile
    drift = 5 * math.sin(time.time() / 300 + self._drift_phase)
    noise = random.gauss(0, p["cpu_variance"] / 2)
    value = p["cpu_base"] + drift + noise
    return round(max(1.0, min(99.0, value)), 1)

Three things are happening here:

The sine wave (math.sin(time.time() / 300 ...)) creates a slow, smooth oscillation with a period of about five minutes. This is the gradual drift you see in real systems as load rises and falls through the day.

The phase offset (self._drift_phase, a random value set once when the device starts) means every device is at a different point in its cycle. Without it, all your simulated devices would drift up and down in perfect unison, which is a dead giveaway that the data is fake.

self._drift_phase = random.uniform(0, 2 * math.pi)

The Gaussian noise (random.gauss) adds small reading-to-reading variation on top of the drift. Real sensors are never perfectly smooth — there's always measurement jitter.

The result is data that drifts, wobbles, and stays within a believable range — and each device has its own personality. When you chart it, it looks like telemetry, not like a random number generator.

Device profiles: a camera is not a sensor

A real fleet isn't 15 copies of the same device. A camera runs hot and busy. A simple sensor sips power and idles. A gateway sits in between. If your simulator treats them all identically, your test data doesn't reflect anything real.

So each device type gets a profile:

DEVICE_PROFILES = {
    "sensor": {
        "cpu_base": 8,    "cpu_variance": 6,
        "temp_base": 42.0, "temp_variance": 5.0,
        "telemetry_interval": 10,
        "failure_rate": 0.03,
    },
    "camera": {
        "cpu_base": 65,   "cpu_variance": 20,
        "temp_base": 61.0, "temp_variance": 10.0,
        "telemetry_interval": 5,
        "failure_rate": 0.02,
    },
    # gateway, controller ...
}

A camera baselines at 65% CPU and 61°C, publishing every 5 seconds. A sensor baselines at 8% CPU and 42°C, publishing every 10 seconds. When this data lands in a dashboard, the device types are visibly different — exactly like a real deployment, where you can often guess a device's role just from its resource profile.

The failure_rate field controls how often the device injects an anomaly — a sudden CPU and temperature spike — so there's something for downstream anomaly detection to actually find:

if self._is_anomaly():
    payload["cpu_percent"] = round(random.uniform(88, 99), 1)
    payload["temperature_c"] = round(random.uniform(78, 92), 2)
    payload["anomaly"] = True

Using robmqtt in the simulator

This is where the simulator doubles as a usage example for the library. Each simulated device is a real robmqtt client:

from robmqtt import ProductionMQTTClient

self.client = ProductionMQTTClient(
    client_id=f"fleet_{device_id}",
    broker_host=broker_host,
    broker_port=broker_port,
    max_queue_size=500,
    db_path=f"./data/{device_id}.db",
    min_backoff=2,
    max_backoff=30,
    log_dir=f"./logs/{device_id}",
)

self.client.connect()
self.client.start()

Each device publishes three kinds of message, and the QoS and priority differ by importance:

# Telemetry — frequent, can tolerate eviction under pressure
self.client.publish(topic=f"fleet/{id}/telemetry",
                    payload=json.dumps(payload), qos=1, priority=5)

# Status — operational health, higher priority
self.client.publish(topic=f"fleet/{id}/status",
                    payload=json.dumps(payload), qos=1, priority=8)

# Boot/alert events — must not be lost, highest priority, QoS 2
self.client.publish(topic=f"fleet/{id}/events",
                    payload=json.dumps(payload), qos=2, priority=10)

This is the priority system from part 1 in action. If the broker goes down and the offline queue fills, routine telemetry (priority 5) gets evicted before status messages (priority 8), and event messages (priority 10, QoS 2) are protected.

The status messages even report the client's own internal state, pulled straight from robmqtt:

stats = self.client.get_statistics()
payload = {
    "queue_depth":     stats.get("offline_queue_size", 0),
    "inflight_count":  stats.get("inflight_count", 0),
    "is_connected":    stats.get("is_connected", False),
    "reconnect_count": stats.get("reconnect_count", 0),
}

So the simulated fleet reports on its own connectivity health — which means you can build a dashboard that shows the offline queue filling and draining in real time.

Watching the offline queue work

This is the part I find satisfying. Start a device:

python device_simulator.py --device-id device_001 --device-type gateway

[device_001] Started — type=gateway location=warehouse_a

Now kill the broker. The device keeps publishing — but the messages are now being written to SQLite instead of sent:

sudo systemctl stop mosquitto

The device doesn't crash. It doesn't error. It just quietly queues. The queue_depth in the status payload climbs: 5, 12, 28, 45...

Bring the broker back:

sudo systemctl start mosquitto

The queue drains automatically. Every reading that piled up during the outage is delivered, in priority order. The queue_depth falls back to zero. Nothing was lost.

That's the whole point of the library, demonstrated in a way you can watch happen.

Why a simulator is worth building

Even if you have real hardware, a simulator earns its place:

It lets you test at scale you don't have hardware for. You can run 15 simulated devices on your laptop and see how your pipeline, database, and dashboards behave under fleet-level load.

It gives you reproducible failure scenarios. Killing a broker on demand is a lot easier than waiting for a real 4G connection to drop in the field.

It produces clean test data with known properties. You injected the anomalies, so you know exactly what your anomaly detection should catch.

And — as a bonus — it doubles as living documentation for how to use your client library. The simulator is the example.

What's next

In part 3 I'll cover running this at fleet scale — launching many devices at once and feeding their telemetry through an analytics pipeline into a live dashboard.

The full simulator code is on GitHub alongside robmqtt:

PyPI: pypi.org/project/robmqtt
GitHub: github.com/ranaweerasupun/resilient-edge-mqtt-client

How do you test your IoT pipelines — real hardware, simulators, or something else? I'm curious what others do — let me know in the comments.

Why Your MQTT Client Is Silently Losing Messages (And How I Fixed It) - robmqtt

Supun Sriyananda — Tue, 26 May 2026 14:01:19 +0000

Why Your MQTT Client Is Silently Losing Messages (And How I Fixed It)

I learned this the hard way.

I was building a sensor system for a field deployment — Raspberry Pi units publishing temperature and humidity data over 4G cellular to an MQTT broker. The dashboard looked fine. The graphs looked fine. Then one day I compared the raw sensor logs against what actually made it to the broker.

Thousands of readings. Gone. No errors. No warnings. Just gone.

The culprit? paho-mqtt's default behaviour when the broker is unreachable: it silently drops your message and moves on.

After losing enough data I wrote a library to fix it. It's now on PyPI as robmqtt.

pip install robmqtt

But before I show you how it works, let me show you exactly what the problem is — because it's subtler than most people realise.

The Problem with Standard MQTT Clients

When you call client.publish() in paho-mqtt and the broker is unreachable, one of two things happens:

The message is silently discarded (QoS 0)
The message is queued in memory for QoS 1/2 — but that queue is lost on process restart, and there's a second gap that even QoS 1 doesn't close

That second gap is the sneaky one. Here's what happens with QoS 1:

The message was "sent" from your perspective. It was never confirmed from the broker's perspective. And paho has no mechanism to track this gap across reconnections.

On a stable data centre network, this almost never matters. On a Raspberry Pi running on 4G cellular in a field cabinet, it happens constantly.

What a Resilient Edge Client Actually Needs

After losing enough data, I sat down and wrote out what a proper edge MQTT client needs to do:

1. Persist offline messages to disk
If the broker is unreachable when publish() is called, the message should be written to disk and replayed later. Not held in memory — memory is lost on restart.

2. Track in-flight messages separately
Messages that have been handed to the broker but not yet ACK'd need to be tracked. On reconnect, they must be re-sent before any queued messages start draining.

3. Priority-based eviction
When the queue fills up, not all messages are equal. A critical alarm should survive. A routine telemetry reading from 6 hours ago should not block it.

4. Exponential backoff on reconnect
A fleet of 50 devices coming back online after a broker restart should not all hammer the broker at the same second.

5. Thread-safe storage
The MQTT network thread and your application thread are both touching message state. This needs to be safe without forcing the caller to think about locking.

None of this is exotic. All of it is missing from the standard paho-mqtt client when used out of the box.

How robmqtt Solves It

Here's the architecture:

The Offline Queue

When the client detects it's disconnected, publish() routes to an OfflineQueue backed by SQLite:

# Simplified from offline_queue.py
class OfflineQueue:
    def enqueue(self, topic, payload, qos, priority):
        with self._lock:
            self._db.execute("""
                INSERT INTO queue (topic, payload, qos, priority, timestamp)
                VALUES (?, ?, ?, ?, ?)
            """, (topic, payload, qos, priority, time.time()))

    def dequeue_batch(self, batch_size=10):
        with self._lock:
            # Highest priority first, then oldest first within same priority
            return self._db.execute("""
                SELECT id, topic, payload, qos FROM queue
                ORDER BY priority DESC, timestamp ASC
                LIMIT ?
            """, (batch_size,)).fetchall()

SQLite gives you durability without a separate process. It survives power cycles. The threading lock means your application thread and the drain thread never step on each other.

The Inflight Tracker

This closes the gap QoS 1 leaves open:

# Simplified from inflight_tracker.py
class InflightTracker:
    def track(self, mid, topic, payload, qos):
        """Call this when you hand a message to paho."""
        with self._lock:
            self._db.execute("""
                INSERT OR REPLACE INTO inflight (mid, topic, payload, qos)
                VALUES (?, ?, ?, ?)
            """, (mid, topic, payload, qos))

    def acknowledge(self, mid):
        """Call this in on_publish callback."""
        with self._lock:
            self._db.execute("DELETE FROM inflight WHERE mid = ?", (mid,))

    def get_all_pending(self):
        """Call this on reconnect — re-send everything unacknowledged."""
        with self._lock:
            return self._db.execute(
                "SELECT topic, payload, qos FROM inflight"
            ).fetchall()

On reconnect, the client replays all inflight messages first, then starts draining the offline queue. Delivery order is preserved.

Priority Eviction

Each message gets a priority from 1 (lowest) to 10 (highest):

# Routine telemetry — can be evicted when queue is full
client.publish(
    topic="sensors/temperature",
    payload='{"value": 23.5}',
    qos=1,
    priority=3
)

# Critical alert — survives eviction, displaces old telemetry
client.publish(
    topic="alerts/critical",
    payload='{"type": "over_temp", "value": 87.2}',
    qos=2,
    priority=9
)

When the queue hits capacity, the lowest-priority messages are evicted first. Your critical alerts are never blocked by a backlog of stale routine data.

Using robmqtt

Install:

pip install robmqtt

Basic usage — this is everything you need:

from robmqtt import ProductionMQTTClient
import json

client = ProductionMQTTClient(
    client_id="field_device_001",
    broker_host="mqtt.yourdomain.com",
    broker_port=1883,
    max_queue_size=5000,    # holds ~5000 messages during outages
    min_backoff=2,          # start retrying after 2s
    max_backoff=60,         # cap retry interval at 60s
    db_path="./device.db",  # SQLite lives here — survives reboots
)

client.connect()
client.start()

# From here just call publish() — routing is handled internally.
# Connected: sends directly and tracks inflight.
# Disconnected: writes to SQLite, drains automatically on reconnect.

while True:
    reading = read_sensor()
    client.publish(
        topic="sensors/temperature",
        payload=json.dumps(reading),
        qos=1,
        priority=5,
    )
    time.sleep(30)

The application code doesn't need to know whether the broker is reachable. That's the point.

Check what's happening at runtime:

stats = client.get_statistics()
print(stats)
# {
#   'is_connected': True,
#   'offline_queue_size': 0,
#   'inflight_count': 2,
#   'reconnect_count': 4,
#   ...
# }

TLS is supported if your broker requires it:

client = ProductionMQTTClient(
    client_id="secure_device_001",
    broker_host="mqtt.yourdomain.com",
    broker_port=8883,
    use_tls=True,
    ca_certs="/etc/ssl/certs/broker-ca.crt",
    username="device001",
    password="your_password",
)

Seeing It in Action

The repo includes test_13.py, a simulation designed specifically to demo the offline behaviour:

# Terminal 1 — run the simulation (publishes every 5 seconds)
python test_13.py

# Terminal 2 — simulate a network outage
sudo systemctl stop mosquitto

# Watch messages queue up in Terminal 1
# Queue stats print every 10 readings

# Restore connectivity
sudo systemctl start mosquitto

# Watch the offline queue drain automatically — zero messages lost

The queue drain happens in a background daemon thread. Your application code does nothing. It just works.

Real-World Context

I've deployed this pattern on:

Battery management systems — monitoring cell voltages and temperatures in production energy storage systems. A 10-minute broker outage during a network switch should not cause a gap in the battery health record.
Robotics telemetry — ROS2 robots publishing sensor and status data. Process restarts during OTA updates should not lose the last known state.
MQTT edge gateways — aggregating data from downstream sensors over serial or CAN and forwarding to a cloud broker over 4G. The gateway may reconnect dozens of times per day.

In all of these cases, the pattern is the same: treat disconnection as normal, not exceptional. Design the client to buffer, not to fail.

Who This Is For

robmqtt is specifically for edge device deployments where:

Network connectivity is unreliable (cellular, Wi-Fi roaming, VPNs)
Process restarts happen (watchdog resets, power cycles, OTA updates)
Message loss has real consequences (industrial monitoring, remote sensors, fleet telemetry)
You don't want to build and maintain this infrastructure yourself

If you're running MQTT on a stable cloud-to-cloud connection, paho-mqtt alone is probably fine. If you're deploying on Raspberry Pi, industrial gateways, field sensors, or anything running on 4G/LTE — this is for you.

What's Next

The Prometheus metrics endpoint is on the roadmap. The structured logging already writes .jsonl metrics files — exposing them via HTTP is a small step and would make robmqtt slot naturally into standard observability stacks.

If you try it and hit an issue, open a GitHub issue. If you want a feature, open a discussion.

PyPI: pypi.org/project/robmqtt
GitHub: github.com/ranaweerasupun/resilient-edge-mqtt-client

Have you run into MQTT message loss on edge devices? How did you handle it — drop a comment below.