DEV Community: Supun Sriyananda

DuckDB vs SQLite: Which one is better?

Supun Sriyananda — Tue, 09 Jun 2026 09:18:52 +0000

I spend most of my time somewhere between microcontrollers and dashboards. One week I'm squeezing firmware onto a device with barely any memory to spare, the next I'm staring at a few million sensor readings trying to work out why a gateway in the field keeps misbehaving. So when people line up "DuckDB vs SQLite" as a fight to the death, I always want to gently jump in.

They're both small. They both run inside your own program instead of off on some server. They both let you work with a single file using SQL. On paper that makes them sound like rivals. But the more I've used them — across embedded work, edge devices, and plain old data crunching — the more they feel like two neighbours who happen to do completely different jobs.

Let me walk through what I mean, starting from the very beginning.

First, what "embedded" actually means here

When people hear the word "database," they often picture something big and separate: a program running on its own server somewhere that your app has to connect to over the network, with a username and password, before it can read or write anything. PostgreSQL and MySQL work like that. There's nothing wrong with it — it's how most large web apps run — but it's a lot of moving parts.

SQLite and DuckDB throw all of that out. There's no separate program to start. Nothing to log into. No server quietly running in the background. The whole database is just an ordinary file sitting on your disk, and the database engine is a small library your code loads in directly. You point it at the file, you run your queries, and that's the whole setup. Nothing else to install or babysit.

That shared simplicity is the lovely part, and it's exactly why you find these two in places a big server-based database could never go — phones, web browsers, tiny sensors, a quick script on your laptop. But the moment you look at how each one stores your data and works through it, they head off in opposite directions. That difference is the whole story, so let's go there next.

SQLite: the one that's already everywhere

SQLite stores data in rows. Picture a spreadsheet where each row is one complete record, and all the values for that record sit together: the id, the name, the temperature reading, all in one place. That layout is perfect when you're constantly poking at individual records — add this new reading, update that setting, grab the latest entry for device 42. You want the whole record at once, and SQLite hands it to you fast.

A few things I love about it:

It's tiny. The whole engine is around a megabyte. On a small device, that matters enormously.
It's everywhere, and I mean everywhere. There are tens of billions of SQLite files in active use — it's running on basically every phone, and your web browser uses it right now to store your history and settings. It has earned the right to be boring.
It's astonishingly reliable. It's one of the most thoroughly tested pieces of software on the planet. The US Library of Congress even recommends SQLite as a format for preserving digital files long-term, because they trust it'll still open decades from now. When you're shipping a device that has to run untouched in a cabinet for years, that track record buys you a lot of peace of mind.

In my embedded work, SQLite is the default for anything that holds state — the current situation the device needs to remember. Its settings. A queue of readings waiting to be uploaded. A small log of recent events. It handles a steady stream of small writes gracefully and it doesn't ask for resources the hardware doesn't have.

Where it starts to struggle is heavy number-crunching across huge piles of data. Ask SQLite to add up and average ten million rows and it'll get there — but slowly, because it reads through the data row by row, dragging along every column even when your question only touches one of them. That's not a flaw. It simply wasn't built for that job.

DuckDB: the one that makes big analysis feel easy

DuckDB flips the storage layout around. Instead of keeping each row together, it keeps each column together — all the temperatures in one place, all the device names in another. So when you ask "what's the average temperature across ten million readings," it reads just the temperature column and skips everything else. On top of that it processes data in big batches rather than one row at a time, which modern processors are very good at. The result is that the heavy questions that make SQLite sweat come back from DuckDB before you've finished a sip of coffee.

There's another part that genuinely changed how I work, and it has nothing to do with speed. DuckDB will read your data files directly, right where they sit:

SELECT device_id, avg(temperature)
FROM 'readings/*.parquet'
GROUP BY device_id;

No loading step. No importing anything into a table first. You point it at a folder of files — CSV, JSON, or Parquet (a compact file format that's common for this kind of data) — and it just reads them. For someone who regularly gets handed a pile of sensor dumps, that's the difference between "give me an afternoon to set up a pipeline" and "give me thirty seconds."

It also handles data bigger than your computer's memory by spilling the overflow to disk, so a dataset that would crash a normal in-memory tool just... works.

It's newer than SQLite and the engine is a bit chunkier, but it's still the same idea at heart: a file and a small library, nothing to run separately.

Are these actually used in the real world?

Both, heavily — this isn't a case of betting on something obscure.

SQLite is the most widely deployed database engine that exists, full stop. The tens of billions of copies in daily use make it more common than every other database combined. It's not going anywhere.

DuckDB is younger but its adoption has shot up. It's pulling in around 37 million downloads a month on Python's package index, it's MIT licensed and free, and it has real commercial backing behind it (a company called MotherDuck builds a cloud service on top while keeping the core engine open and free), which answers the usual worry about whether an open-source tool will still be maintained in five years. It also fits neatly with where the industry is heading: open file formats, modern multi-core processors, and even AI coding assistants, which tend to be good at writing SQL and so reach for DuckDB naturally.

And SQLite isn't standing still either. Newer spin-offs like Turso/libSQL are adding things like replication and edge support on top of the classic engine. Both of these tools are safe bets.

So when does each one actually win?

Here's how it shakes out across the three places I work.

On small, constrained hardware, SQLite, almost every time. DuckDB's appetite for memory and processing power during a big query is more than a tiny chip wants to give. On a larger edge device running Linux it's a different story, but down at the small end, SQLite's tiny size and long history are hard to argue with.

At the edge — meaning the gateway boxes that sit between your sensors and the cloud — they actually team up. A pattern I keep coming back to: let SQLite handle the incoming readings on the device, quietly buffering them as they arrive, then let DuckDB do the local number-crunching before anything gets sent upstream. You ship neat summaries instead of the raw firehose, which is kinder to both your bandwidth bill and your cloud costs. They're not competing here. They're a relay team.

For sitting down and analysing data, DuckDB is the one I reach for. Being able to throw SQL at a folder of files without setting up any heavy machinery is exactly the kind of low-fuss tool the job usually calls for. It has quietly become a go-to for local analysis, and for good reason.

The one line I keep in my head

If I had to shrink all of this down to a fridge magnet:

SQLite is for managing data while it's being created and changed. DuckDB is for analysing it once it's all piled up.

Same small, no-server spirit — opposite ends of the data's life. SQLite looks after the data as it's coming in and changing. DuckDB shows up later to make sense of the whole pile. Once that clicked for me, the "versus" framing kind of fell apart.

So if you're choosing between them, the honest answer is often both, for different things. Work out whether the job in front of you is about handling data as it changes or making sense of a big pile of it, and the choice mostly makes itself.

If you've wired these two together in your own projects, I'd love to hear how you split the work. I'm always tinkering with my own setup, and the edge folks always seem to have the most interesting war stories.

Deploying Production Systems on Raspberry Pi: Lessons from the Field

Supun Sriyananda — Sun, 07 Jun 2026 05:38:31 +0000

Deploying Production Systems on Raspberry Pi: Lessons from the Field

These are the things I wish I had known before deploying Pis in production.

SD Cards Will Kill You

The first production Pi I deployed used a generic microSD card. It failed after four months. The second one used a "name brand" card. It failed after six months. The pattern remained always the same: the filesystem corrupts during a power loss, the Pi boots into read-only mode, and whatever the system was supposed to be doing silently stops working.

SD card corruption under power loss is not a bug you can fix in software. It is a fundamental characteristic of flash storage that was designed for cameras, not servers. The cells wear out, write operations are not atomic, and a sudden power cut mid-write leaves the filesystem in a state that fsck (File System Consistency Check) sometimes cannot recover.

If you choose SD cards: Switch to a Pi-rated industrial SD card (SanDisk MAX Endurance, Samsung Pro Endurance) or eliminate the SD card entirely by booting from a USB SSD. USB boot on Pi 4 and Pi 5 is stable and the endurance difference is enormous — a decent SSD handles orders of magnitude more write cycles than any SD card.

For systems that must use SD, mount the filesystem read-only and put all writable state on a tmpfs or a separate partition with journaling.

tmpfs is a special type of temporary file storage facility in Linux and Unix-like systems that stores files directly in volatile memory (RAM) instead of on a persistent drive like an SD card or SSD.

When you mount a folder as tmpfs, any files written to that folder behave like regular files, but they consume RAM and exist purely in memory.

# /etc/fstab — mount root read-only, put logs and data elsewhere
/dev/mmcblk0p2  /        ext4  ro,defaults  0  1
tmpfs           /tmp     tmpfs defaults      0  0
tmpfs           /var/log tmpfs defaults      0  0

# Separate partition for application data with journaling
/dev/mmcblk0p3  /var/lib/myapp  ext4  defaults,data=journal  0  2

For a warehouse sensor deployment, the application writes data to SQLite on a separate ext4 partition with journaling enabled. If the Pi loses power mid-write, fsck can recover the journal. The root partition is read-only and survives power loss cleanly every time.

Thermal Throttling Is Silent and Intermittent

The thing is Pi will not tell you that it is throttling. It will not log a warning. That means, your video stream will just start dropping frames, your serial latency will increase, and your MQTT reconnects will take longer. And as you can see, all these symptoms look like software bugs. But if you check:

vcgencmd get_throttled

vcgencmd get_throttled is a command line tool unique to the Raspberry Pi that checks whether the computer has lowered its CPU speed.

0x0 means everything is fine. Anything else means the Pi is throttling now or has throttled since last boot. The common values:

0x50005 — The danger zone. You are currently under-volted and throttled right now, and it has happened before.
0x50000 — This means your system is physically running fine right now, but under-voltage (0x10000) and throttling (0x40000) have occurred in the past. Your current power supply is dropping voltage under load, making your SD card highly vulnerable to corruption.
0x4 — soft temperature limit active

I was building a telepresence robot, encoding 720p30 H.264 while running the aiohttp server and serial communication was pushing the Pi 4 to 80°C without a heatsink. The encoder started dropping frames randomly. Adding a heatsink brought idle temperature to 45°C and load temperature to 62°C. Managed to get the throttling under control.

On a Pi 5, the situation is better but not solved. The Pi 5 has an active cooler as an official accessory and it is worth using in any deployment where the Pi is in an enclosure. But, enclosures trap heat. A Pi in a plastic project box with no airflow will throttle faster than a bare board.

Also it is a best practice to add temperature monitoring to your health check endpoint so you find out before users do:

import subprocess

def get_pi_health():
    temp = subprocess.run(
        ['vcgencmd', 'measure_temp'],
        capture_output=True, text=True
    ).stdout.strip()  # "temp=58.0'C"

    throttled = subprocess.run(
        ['vcgencmd', 'get_throttled'],
        capture_output=True, text=True
    ).stdout.strip()  # "throttled=0x0"

    return {
        'temperature': temp,
        'throttled': throttled,
        'throttled_ok': throttled == 'throttled=0x0'
    }

Power Supply Quality Matters More Than Rated Current

A power supply rated for 3A is not always a 3A power supply. Cheap USB-C supplies have poor voltage regulation. So, under a load they sag below the 5.1V the Pi needs and trigger under-voltage throttling. The symptom is identical to thermal throttling: random slowdowns, occasional reboots, SD card corruption on shutdown.

The official Raspberry Pi power supply is not a premium product — it is a specification-compliant one. Use it, or use a bench power supply for development and a known-good supply for deployment. The Pi 5 draws up to 5A under full load; a 3A supply will cause under-voltage events when the CPU is fully loaded.

For any deployment running off mains power, a UPS hat (Geekworm UPS hat, Waveshare UPS hat) is worth the £20. The Pi gets notified of incoming power loss and can initiate a clean shutdown before the battery dies, which eliminates the entire class of "power cut during SD write" corruption events.

Network time synchronization errors can cause unexpected system failures

A Pi that has been offline for an extended period will have a wrong system clock when it boots — sometimes wrong by days if the RTC battery is dead or there is no RTC at all. Applications that timestamp log entries, certificate validity checks, and SQLite timestamp comparisons all behave unexpectedly when the system time is wrong.

A specific failure case I encountered: the MQTT client was writing timestamps using datetime.now().isoformat(). After a boot without internet, the system clock was set to 2023-01-01 (the default). All queued messages got timestamps in 2023. When the clock corrected to 2024 via NTP after network connection, the retention policy deleted those messages as being "older than 7 days" — because relative to the current time they appeared to be a year old.

Fix 1: Use a hardware RTC. The DS3231 costs about £3 and keeps accurate time across power cycles without network. Enable it with a device tree overlay:

# /boot/firmware/config.txt
dtoverlay=i2c-rtc,ds3231

Fix 2: For timestamps that survive offline periods, use monotonic time for intervals and NTP-synced time only for absolute timestamps. Do not mix them.

Fix 3: Configure chrony or systemd-timesyncd to be aggressive about syncing on boot and to accept large time jumps:

# /etc/chrony.conf
makestep 1 -1   # Accept any step size, any number of times

Watchdog Timers Are Not Optional

If your Pi is in a location where you cannot physically reach it — mounted on a robot, installed in a warehouse, bolted inside a wall panel — a software crash or infinite loop that freezes the application is effectively a permanent failure until someone intervenes.

The Linux kernel watchdog kicks the hardware watchdog timer while the kernel is running. If the kernel hangs, the watchdog expires and forces a reboot. But it does not know whether your application is running correctly. For that, you need an application-level watchdog.

systemd's built-in watchdog support requires almost no code:

# In your main loop, notify systemd you're still alive
import os
import time

def notify_watchdog():
    """Tell systemd the application is healthy."""
    os.system("systemd-notify WATCHDOG=1")

# Call this from your main loop — if you stop calling it,
# systemd will restart the service
while True:
    do_work()
    notify_watchdog()
    time.sleep(5)

# /etc/systemd/system/myapp.service
[Service]
WatchdogSec=30        # Restart if no heartbeat for 30 seconds
Restart=always
RestartSec=5
StartLimitIntervalSec=120
StartLimitBurst=5

For applications where notify_watchdog() cannot be called from the main loop (async applications, multi-threaded servers), run it from a background thread that monitors the health of the main thread.

Remote Access Must Work Before Anything Breaks

The time to set up remote access is before you deploy, not after. I use Tailscale for most Pi deployments because it takes five minutes to configure, works through NAT, does not require port forwarding, and uses WireGuard under the hood. Once it is running, you have a reliable backdoor to your hardware. However, if you need more control, complete data sovereignty, and zero third-party dependencies, use vanilla WireGuard instead. While WireGuard requires you to manually configure routing rules and host a central server with an open port for NAT traversal, it gives you total ownership over your network topology without device or account limitations:

# Install
curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up

# Enable SSH in your tailnet policy and you can reach the Pi from anywhere
ssh user@cyrobot.turkey-trench.ts.net

Tailscale also provides valid TLS certificates for your device's hostname in the tailnet — which is how the WebRTC server serves HTTPS without a domain name or public CA.

Set up mosh alongside SSH for unreliable connections. Regular SSH sessions die when the network hiccups. Mosh sessions survive.

Logs Fill Up the Filesystem

/var/log on a production Pi will fill up over months of continuous operation. When it does, your application cannot write logs, SQLite cannot open its WAL file, and things fail in confusing ways that do not obviously point to "disk full."

Set up log rotation from day one:

# /etc/logrotate.d/myapp
/var/log/myapp/*.log {
    daily
    rotate 7
    compress
    missingok
    notifempty
    size 10M
}

And add disk usage to your health check:

import shutil

def check_disk():
    usage = shutil.disk_usage('/')
    free_percent = (usage.free / usage.total) * 100
    return {
        'free_gb': round(usage.free / 1e9, 2),
        'used_percent': round(100 - free_percent, 1),
        'warning': free_percent < 15
    }

The Summary

Before deploying any Pi to a location you cannot easily reach:

Boot storage is an industrial SD card or USB SSD. The root filesystem is read-only or has journaling on writable partitions. A hardware RTC is installed. A heatsink or active cooler is fitted. A quality power supply is used and a UPS hat is fitted if on mains power. Tailscale is installed and tested from outside the local network. systemd service has Restart=always, WatchdogSec, and StartLimitBurst set. Log rotation is configured. A health endpoint exposes temperature, throttle status, and disk usage. You have confirmed you can SSH in and restart the service from the office before going to the deployment site.

Written by Supun Akalanka | Category: Lessons Learned | Tags: Raspberry Pi, Production, Reliability, Embedded Linux, Hardware

Creating Robust systemd Services for Embedded Applications

Supun Sriyananda — Wed, 03 Jun 2026 09:51:04 +0000

There is a moment every embedded Linux developer hits eventually. You have spent days building something that works beautifully — a sensor pipeline, a streaming server, an MQTT client — and then you reboot the device and everything is silent. Nothing started. You SSH in, manually run your script, and it all comes back to life. The hardware is fine. Your code is fine. You just have no way of automatically running it.

That is the gap systemd fills. It is the init system on virtually every modern Linux distribution, and on embedded Linux systems like the Raspberry Pi it is what decides what runs at boot, what gets restarted if it crashes, and where all the logs go. Once you understand how to write a service file, your applications stop being fragile scripts you need to babysit and start being first-class system services that survive reboots, network drops, and unexpected crashes.

This tutorial builds up from the simplest possible service file to a production-ready configuration, explaining every line along the way. By the end you will have a service running your own Python application, logging to the system journal, and automatically restarting itself after failures.

See Complete Tutorial in Github: Systemd Services Tutorial

What systemd Actually Does

Before writing any configuration, it helps to understand what problem systemd is solving, because the design of service files makes much more sense once you see the underlying model.

When your Raspberry Pi boots, the Linux kernel starts and immediately hands control to process ID 1 — the very first user-space process. On modern systems, that process is systemd. Everything that happens next — mounting filesystems, bringing up the network, starting your application — is orchestrated by systemd. It reads configuration files called unit files that describe what should be started, when, in what order, and what to do if something goes wrong.

A service file is just one type of unit file (there are also unit files for timers, sockets, mount points, and more, but services are what you will use most). When you tell systemd about your application through a service file, you are essentially saying: "here is my program, here is when I want it to run, and here is how I want you to manage it." systemd takes it from there — starting it at boot, watching it, restarting it if it dies, and capturing everything it prints to stdout and stderr into a structured log called the journal.

The central tool for interacting with systemd is systemctl. You use it to start, stop, enable, disable, and inspect services. The companion tool journalctl gives you access to the journal — the logs that systemd collects from every service it manages.

A Minimal Service File

Let us start with the simplest possible service to understand the structure, then build from there. Suppose you have a Python script at /home/pi/mqtt_client/production_client.py that you want to run automatically at boot.

Service files live in /etc/systemd/system/. Create a new one:

sudo nano /etc/systemd/system/mqtt-client.service

Here is the minimal version:

[Unit]
Description=Production MQTT Edge Client

[Service]
ExecStart=/usr/bin/python3 /home/pi/mqtt_client/production_client.py

[Install]
WantedBy=multi-user.target

Even this tiny file has three sections and each one has a specific job. The [Unit] section contains metadata and dependency declarations — the Description is just a human-readable label that appears in logs and status output. The [Service] section is where the actual execution configuration lives — right now we only have ExecStart, which is the command that launches your program. The [Install] section controls how the service integrates into the boot process — WantedBy=multi-user.target means "start this service when the system reaches the normal multi-user state", which is essentially "start this at boot when the system is ready for normal operation."

To activate it, you need to do two things: reload systemd so it picks up the new file, and enable the service so it starts at boot:

# Tell systemd to re-read all unit files
sudo systemctl daemon-reload

# Enable it to start at boot
sudo systemctl enable mqtt-client.service

# Start it right now without waiting for a reboot
sudo systemctl start mqtt-client.service

Check whether it is running:

sudo systemctl status mqtt-client.service

You will see output that tells you the service state active (running). This is what you want, the process ID, when it started, and the last few lines of log output. If it failed, the status output will usually tell you exactly why — an incorrect path, a Python import error, whatever the problem is.

The difference between enable and start trips people up at first, so it is worth being explicit: enable creates a symlink that tells systemd to start the service at boot — it does not start it right now. start starts it immediately — it does not persist across reboots. In practice you almost always want both.

Making it Actually Robust: Restart Policies

The minimal service above will start your application at boot, but if the application crashes — which happens, especially in edge environments with unreliable hardware or network — systemd will not do anything. It will just leave the service in a failed state. For an embedded device running unattended in the field, that is not acceptable.

The Restart directive in the [Service] section tells systemd what to do when your application exits. Here is what the options mean in practice. no means never restart (the default — not what you want for production). on-failure means restart if the process exits with a non-zero exit code or is killed by a signal, but not if it exits cleanly with code 0. always means restart no matter what — even if the process exits with code 0. on-abnormal means restart on crash or signal, but not on clean exit or timeout.

For most embedded applications, on-failure is the right choice. It means "if something goes wrong and the program dies unexpectedly, bring it back", but it also means "if I deliberately stop the service with systemctl stop, do not restart it."

There is one more important setting to pair with Restart: RestartSec, which sets how long to wait before restarting. Without it, systemd restarts immediately, which can cause problems if your service is crashing due to a dependency that is not ready yet (like the network). A short delay — even two or three seconds — gives things time to settle.

Here is the updated service with restart policy:

[Unit]
Description=Production MQTT Edge Client

[Service]
ExecStart=/usr/bin/python3 /home/pi/mqtt_client/production_client.py
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

Declaring Dependencies

This is where systemd really starts to earn its keep. Most embedded applications do not run in isolation — they depend on the network being up, a filesystem being mounted, or another service being ready. systemd lets you express these dependencies explicitly so that your service starts at the right time and in the right order.

The two most commonly confused directives here are After and Requires. Think of them as answering different questions. After=network.target answers the question "when should I start?" — it tells systemd to not even attempt to launch this service until the network target has been reached. Requires=network.target answers the question "what must exist for me to function?" — it tells systemd that if the network goes away, this service should be stopped too.

You can use them together, and for network-dependent applications you almost always should. There is also Wants, which is a softer version of Requires — it expresses preference rather than hard dependency. If what Wants declares is not available, the service will still start rather than failing.

For an MQTT client that needs the network to do anything useful:

[Unit]
Description=Production MQTT Edge Client
After=network.target
Wants=network.target

[Service]
ExecStart=/usr/bin/python3 /home/pi/mqtt_client/production_client.py
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

Wants rather than Requires here is deliberate. The MQTT client already handles network loss internally — it has exponential backoff and an offline queue for exactly this scenario. So we do not want systemd to kill the service if the network drops; we just want to make sure we do not start before the network stack is even initialised.

For cases where your application would be completely broken without a dependency — a database-backed service where the database is also managed by systemd, for example — use Requires. The distinction matters in practice.

Working Directory and Environment

Two things that bite almost every developer on their first real systemd deployment are working directory and environment variables. When you run a script manually from your terminal, you have a working directory (usually your home folder) and a full set of environment variables inherited from your shell. systemd does not give you either of those by default.

This matters because Python scripts often use relative paths like ./config.json or ./logs/, and those paths will fail when the working directory is not what you expect. Environment variables like PATH, HOME, and any custom variables your application reads from the environment will also be missing or wrong.

WorkingDirectory solves the path problem, and Environment or EnvironmentFile solves the variable problem:

[Unit]
Description=Production MQTT Edge Client
After=network.target
Wants=network.target

[Service]
# Set the working directory so relative paths work correctly
WorkingDirectory=/home/pi/mqtt_client

# Pass environment variables directly
Environment=MQTT_BROKER_HOST=192.168.1.100
Environment=MQTT_CLIENT_ID=warehouse_sensor_01

# Or load them from a file (better for secrets)
# EnvironmentFile=/etc/mqtt-client/config.env

ExecStart=/usr/bin/python3 /home/pi/mqtt_client/production_client.py
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

The EnvironmentFile approach is worth knowing about even if you do not use it immediately. It lets you store configuration in a separate file — including secrets like passwords — outside the service file itself. A file like /etc/mqtt-client/config.env contains simple KEY=value lines and is loaded by systemd before starting the service. This keeps credentials out of version control and makes configuration changes possible without touching the service file.

Logging

One of systemd's genuine gifts to embedded developers is centralised logging. Everything your application prints to stdout and stderr is automatically captured by the journal — no log file configuration required, no rotation to set up. The journal is structured, persistent across reboots (on most systems), and queryable in powerful ways.

The basic command to read logs from your service is:

# Show all logs from your service
journalctl -u mqtt-client.service

# Follow logs in real-time (like tail -f)
journalctl -u mqtt-client.service -f

# Show only logs since the last boot
journalctl -u mqtt-client.service -b

# Show logs from the last hour
journalctl -u mqtt-client.service --since "1 hour ago"

For embedded devices with limited storage, it is worth explicitly configuring how much disk space the journal is allowed to use. You do this not in the service file but in the journal configuration:

sudo nano /etc/systemd/journald.conf

Add or modify these lines:

[Journal]
# Maximum journal size on disk
SystemMaxUse=50M

# Maximum size of a single journal file
SystemMaxFileSize=10M

Then restart the journal daemon:

sudo systemctl restart systemd-journald

For embedded deployments where the device has a small SD card or eMMC, keeping the journal small is important. Fifty megabytes is a reasonable limit for most edge devices.

A Production-Ready Example

Putting all of this together, here is what the service file looks like for a real deployment — the kind of configuration you would use for a long-running embedded application that needs to be reliable in the field. This is modelled after exactly how I deploy the MQTT edge client and the WebRTC streaming service on the telepresence robot:

[Unit]
Description=Production MQTT Edge Client with Offline Resilience
# Human-readable detail shown in status output and logs
Documentation=https://github.com/ranaweerasupun/mqtt-production-client

# Start after the network is available, but do not require it —
# the application handles network loss internally with its own
# backoff and offline queue
After=network.target
Wants=network.target

# If the service restarts more than 5 times in 60 seconds, give up
# This prevents an infinite crash-restart-crash loop from overwhelming the system
# (these belong in [Unit] on systemd v230+)
StartLimitIntervalSec=60
StartLimitBurst=5

[Service]
# Run as a specific user rather than root — better security practice
User=pi
Group=pi

# Set working directory so relative paths in the application work
WorkingDirectory=/home/pi/mqtt_client

# Load configuration from a separate file
# This keeps secrets out of the service file
EnvironmentFile=/etc/mqtt-client/config.env

# The command that starts the application
ExecStart=/usr/bin/python3 /home/pi/mqtt_client/production_client.py

# What to do when the process exits unexpectedly
# on-failure: restart if the process crashes or exits non-zero
# but NOT if you run 'systemctl stop mqtt-client'
Restart=on-failure

# Wait 5 seconds before restarting
# This prevents rapid restart loops if something is fundamentally broken
RestartSec=5

# Give the service up to 30 seconds to stop gracefully before killing it
TimeoutStopSec=30

# Capture stdout and stderr to the journal
StandardOutput=journal
StandardError=journal

# Tag journal entries with this identifier for easy filtering
SyslogIdentifier=mqtt-client

[Install]
WantedBy=multi-user.target

The StartLimitIntervalSec and StartLimitBurst combination is worth understanding because it solves a real problem. Imagine your MQTT client has a bug that makes it crash immediately on startup — perhaps a malformed config file or a missing dependency. Without these limits, systemd would restart it immediately, it would crash again, systemd would restart it again, and this would loop forever, consuming CPU and filling your journal with crash logs. With StartLimitBurst=5 and StartLimitIntervalSec=60, systemd will make five restart attempts within a 60-second window, and if all five fail, it marks the service as failed and stops trying. At that point systemctl status will clearly tell you the service has hit its restart limit, which prompts you to actually investigate the root cause.

The User=pi directive is also important for production embedded deployments. Running services as root is a security risk — if your MQTT client has a vulnerability, an attacker who exploits it gets root access. Running as a non-privileged user limits the damage. The tradeoff is that you need to make sure that user has permission to access the files and ports your application needs.

Useful Commands to Know

Once your service is running, a handful of commands will cover most of what you need day-to-day:

# Check current status, recent logs, and whether it is enabled at boot
sudo systemctl status mqtt-client.service

# Start the service right now
sudo systemctl start mqtt-client.service

# Stop the service (will not restart automatically due to Restart=on-failure)
sudo systemctl stop mqtt-client.service

# Restart the service (useful after changing your application code)
sudo systemctl restart mqtt-client.service

# Reload the service file after you edit it (then restart to apply changes)
sudo systemctl daemon-reload
sudo systemctl restart mqtt-client.service

# Disable the service from starting at boot
sudo systemctl disable mqtt-client.service

# View full log history for this service
journalctl -u mqtt-client.service

# View logs in real-time
journalctl -u mqtt-client.service -f

One workflow note: whenever you edit the service file, you must run daemon-reload before the changes take effect. systemd caches unit file contents and will not notice your edits otherwise. Forgetting this step is a common source of confusion when your changes do not seem to be working.

What systemd Gives You for Free

It is worth pausing to appreciate what you get from writing a proper service file, because the alternative — ad hoc startup scripts in /etc/rc.local or cron @reboot jobs — gives you almost none of it.

With a systemd service you get automatic restart on crash, ordered startup relative to other system components, centralised and queryable logging, a clean mechanism to start and stop the application during development, restart rate limiting to prevent crash loops, graceful shutdown handling, and the ability to run as a non-root user easily. All of that from a text file that is, at its core, less than twenty lines.

For an embedded device deployed in a factory, a warehouse, or anywhere else it needs to run unattended for months at a time, that reliability infrastructure is not optional. systemd gives it to you essentially for free, as long as you take the time to describe your service correctly.

Finishing What I Started — From a TODO List to a Published PyPI Package

Supun Sriyananda — Thu, 28 May 2026 13:03:03 +0000

This is a submission for the GitHub Finish-Up-A-Thon Challenge

What I Built

I built robmqtt — a resilient MQTT client for edge IoT devices, now published on PyPI.

It came out of a problem that cost me real time. I deploy sensor systems on hardware that lives in the field: Raspberry Pi units on 4G cellular, embedded gateways, battery monitoring systems. On those networks, connectivity is never stable. And the standard MQTT client, paho-mqtt, silently drops messages when the broker is unreachable — no error, no warning, no log entry unless you write one yourself.

I lost data for a long time before I understood why. And finding the cause was brutal. There are no proper logs unless you explicitly add them, so I spent hours — days — chasing a problem that left no trace. I went through forums, blogs, chat AIs, everywhere, looking for how other people had solved it. What I found was the same thing every time: people asking for a production-resilient MQTT client, and no real answer. Just scattered advice and code snippets that handled one piece and ignored the rest.

So I built it myself:

Offline queue — when the broker is unreachable, messages are written to SQLite instead of being dropped. They survive process restarts and power cycles.
Inflight tracking — messages sent but not yet acknowledged are tracked separately and re-sent on reconnect, closing a gap that even QoS 1 leaves open.
Priority eviction — when the queue fills, low-priority telemetry is evicted before critical alerts.
Exponential backoff — a fleet of devices reconnecting after an outage won't all hammer the broker at once.
TLS support — for brokers that require encrypted connections.

And then everything a library needs to actually be used in production, not just by me:

TLS and mutual TLS plus username/password auth, for secured brokers.
Bidirectional messaging — subscribe/unsubscribe with full MQTT wildcard support, and subscriptions that survive reconnects.
Observability — a built-in HTTP health check endpoint (/health returning healthy / degraded / unhealthy) with Docker HEALTHCHECK and Kubernetes liveness-probe examples.
Structured logging throughout — so the next person doesn't lose days to a problem with no trace, the way I did.
73 tests across the storage, tracking, topic-matching, and client layers.

pip install robmqtt

This matters to me because it's not a toy. It's running on real systems I maintain — battery management monitoring and robotics telemetry — where a gap in the data record has actual consequences.

Demo

PyPI: https://pypi.org/project/robmqtt/
GitHub: https://github.com/ranaweerasupun/resilient-edge-mqtt-client

The most satisfying way to see it work is to watch the offline queue survive a broker outage. The repo includes a simulation script for exactly this:

# Terminal 1 — run a simulated device publishing every 5 seconds
python test_13.py

# Terminal 2 — kill the broker mid-run
sudo systemctl stop mosquitto

The device keeps publishing. Messages start queuing to SQLite instead of erroring. Then:

# Bring the broker back
sudo systemctl start mosquitto

The queue drains automatically, in priority order. Every message that piled up during the outage is delivered. Nothing is lost.

Basic usage looks like this — the application code never has to know whether the broker is reachable:

from robmqtt import ProductionMQTTClient
import json, time

client = ProductionMQTTClient(
    client_id="field_device_001",
    broker_host="mqtt.yourdomain.com",
    broker_port=1883,
    max_queue_size=5000,
    db_path="./device.db",
)
client.connect()
client.start()

while True:
    client.publish(
        topic="sensors/temperature",
        payload=json.dumps(read_sensor()),
        qos=1,
        priority=5,
    )
    time.sleep(30)

The Comeback Story

Here's the honest before-and-after.

Before. Once I'd finally figured out the code that solved my problem, I used it — and it went into my GitHub repo, where it sat. It worked. It solved the thing that had cost me days. But no one knew it existed.

And that was the part that nagged at me. I'd keep seeing people online asking for exactly what I'd built — something resilient enough for production — and getting the same non-answers I'd gotten: random advice, half-solutions, snippets. I had the answer sitting in a corner of my repo, and a thought stuck in the corner of my head: it's right here. I need to show this off.

The things I knew I should do but hadn't:

Package it for PyPI so people could pip install it instead of cloning the repo
Add TLS/mTLS support for secure broker connections
Expose a metrics endpoint for observability

It was the classic unfinished side project. Functional, but not shipped. Not something anyone other than me could actually use. The plan was always "publish it once it's polished" — and polishing it was the part that kept not happening.

The hard part. I have a full-time job. Finishing this wasn't something I could do in work hours. So it happened on weekends, late at night, and in the gaps — I drew flowcharts and planned the package structure on my commute, in my head, on paper. Progress came in small pieces, squeezed around everything else. There were plenty of stretches where it would have been easier to just leave it in the repo and move on.

After. I pushed it over the line:

Published to PyPI as robmqtt v1.0.0. This was more work than I expected — restructuring the code into a proper installable package, writing the pyproject.toml, sorting out the module layout, testing the install in a clean environment.
Added TLS support. The client now takes use_tls, ca_certs, certfile, keyfile, and broker auth parameters, so it works with secured brokers, not just local plaintext ones.
Observability Built an HTTP health check endpoint with Docker and Kubernetes probe examples — so the library reports its own health instead of failing silently.
Structured logging so the next person doesn't suffer days to a problem with no trace, the way I did!
Data Integrity Fixed three bugs I'm glad I caught before anyone relied on it: binary payloads being corrupted by string conversion, a resend-tracking gap that could silently lose messages on a second disconnect, and a race condition in publish() that could drop a message between the connection check and the send.
Started building a real project on top of it. To put the library through its paces in a full system, I'm building an IoT fleet analytics platform around it — simulated devices publishing through robmqtt, an MQTT-to-InfluxDB bridge, Grafana dashboards, and statistical anomaly detection. It's still in progress, but it's already doing what mattered most: proving the library holds up as the foundation of a real pipeline, not just in isolated tests.

When it was finally published and I ran the test scripts and watched the offline queue fill and drain exactly the way it was supposed to — messages held through an outage, then delivered, nothing lost — there was nothing more rewarding. The thing that had cost me days of frustration was now one pip install away for anyone who hits the same wall I did.

My Experience with GitHub Copilot

I'll be honest about my workflow, because the reason I relied on Copilot is specific.

I use a few AI tools when I build things. The general-purpose chat assistants are useful for sketching out an approach or talking through a design — but when it came to actual code, they kept giving me snippets that didn't quite work. That's not surprising: they can't see my project. They don't know my file structure, my variable names, the exact version of a library I'm using, or how the piece they're suggesting fits the rest of the code. So I'd paste in a snippet, hit an error, paste the error back, get a revised snippet, hit another error. Going back and forth with a chat window that can't see my codebase got slow and frustrating.

Copilot was different because it lives in my editor and can see everything — all my open files, the actual code around the cursor, the real context. That's why it became the tool I leaned on.

Two ways I used it:

Inline completion, always on. As I typed, Copilot autocompleted the repetitive, structural parts — the device profile dictionaries, the JSON payload construction, the boilerplate around publish() calls with their QoS and priority arguments. Because it could see the patterns already in my file, its suggestions actually fit, instead of being generically plausible code I'd have to rewrite.

Copilot Chat for debugging in context. This is where it earned its place. When something broke, I'd ask it directly — "Why am I getting this error, and how can I fix it?" or "Can you check this code snippet?" — and because it could read my actual files, the answers were grounded in my real code, not a guess about what my code might look like. That's the difference that made me stop pasting snippets into external chat windows. The tool that can see your codebase gives you answers that apply to your codebase.

That freed me to spend my real thinking time on the parts that mattered: the sine-wave drift model for realistic sensor data, the priority eviction logic, and the inflight-tracking design that closes the QoS 1 gap. The hard design decisions were mine. Copilot removed the friction around them — the typing, the boilerplate, and the slow debugging loop — so I could stay focused on the actual engineering.

Built by Supun Sriyananda — R&D Engineer working on embedded and IoT systems. robmqtt is open source on GitHub and PyPI.

How to Generate Realistic IoT Sensor Data for Testing Your MQTT Pipeline

Supun Sriyananda — Thu, 28 May 2026 11:02:48 +0000

How to Generate Realistic IoT Sensor Data for Testing Your MQTT Pipeline

This is part 2 of a series on building robmqtt. Part 1 covered why paho-mqtt silently drops messages and the library I built to fix it. This part is about testing — how to exercise an MQTT pipeline without deploying physical hardware.

In the last post I wrote about robmqtt, a resilient MQTT client for edge devices. Once I had it working, I hit the obvious next problem: how do I test it properly without setting up a rack of Raspberry Pis?

I needed data flowing through the pipeline. Lots of it. From many devices. Behaving differently. And ideally surviving a broker outage so I could watch the offline queue do its job.

So I wrote a device simulator. And in writing it, I learned that generating realistic fake sensor data is harder than it looks — and that most people get it wrong in the same way.

The trap: random data isn't realistic data

The first instinct when simulating a sensor is to reach for random.uniform():

cpu = random.uniform(0, 100)   # don't do this

This produces data that looks nothing like a real sensor. Real sensor readings don't jump randomly across the whole range every second. A CPU sitting at 8% doesn't suddenly read 94% then drop to 3%. Temperature drifts slowly. Signal strength wobbles around a baseline. There's continuity from one reading to the next.

If you test your pipeline with pure random noise, your charts look like static, your anomaly detection has nothing meaningful to detect, and your dashboards are useless for spotting whether anything actually works.

I wanted data that looked like it came from a real device.

Realistic drift with a sine wave

The trick I landed on was a slow sine wave with per-device random phase, plus a little Gaussian noise on top:

def _cpu(self) -> float:
    p = self.profile
    drift = 5 * math.sin(time.time() / 300 + self._drift_phase)
    noise = random.gauss(0, p["cpu_variance"] / 2)
    value = p["cpu_base"] + drift + noise
    return round(max(1.0, min(99.0, value)), 1)

Three things are happening here:

The sine wave (math.sin(time.time() / 300 ...)) creates a slow, smooth oscillation with a period of about five minutes. This is the gradual drift you see in real systems as load rises and falls through the day.

The phase offset (self._drift_phase, a random value set once when the device starts) means every device is at a different point in its cycle. Without it, all your simulated devices would drift up and down in perfect unison, which is a dead giveaway that the data is fake.

self._drift_phase = random.uniform(0, 2 * math.pi)

The Gaussian noise (random.gauss) adds small reading-to-reading variation on top of the drift. Real sensors are never perfectly smooth — there's always measurement jitter.

The result is data that drifts, wobbles, and stays within a believable range — and each device has its own personality. When you chart it, it looks like telemetry, not like a random number generator.

Device profiles: a camera is not a sensor

A real fleet isn't 15 copies of the same device. A camera runs hot and busy. A simple sensor sips power and idles. A gateway sits in between. If your simulator treats them all identically, your test data doesn't reflect anything real.

So each device type gets a profile:

DEVICE_PROFILES = {
    "sensor": {
        "cpu_base": 8,    "cpu_variance": 6,
        "temp_base": 42.0, "temp_variance": 5.0,
        "telemetry_interval": 10,
        "failure_rate": 0.03,
    },
    "camera": {
        "cpu_base": 65,   "cpu_variance": 20,
        "temp_base": 61.0, "temp_variance": 10.0,
        "telemetry_interval": 5,
        "failure_rate": 0.02,
    },
    # gateway, controller ...
}

A camera baselines at 65% CPU and 61°C, publishing every 5 seconds. A sensor baselines at 8% CPU and 42°C, publishing every 10 seconds. When this data lands in a dashboard, the device types are visibly different — exactly like a real deployment, where you can often guess a device's role just from its resource profile.

The failure_rate field controls how often the device injects an anomaly — a sudden CPU and temperature spike — so there's something for downstream anomaly detection to actually find:

if self._is_anomaly():
    payload["cpu_percent"] = round(random.uniform(88, 99), 1)
    payload["temperature_c"] = round(random.uniform(78, 92), 2)
    payload["anomaly"] = True

Using robmqtt in the simulator

This is where the simulator doubles as a usage example for the library. Each simulated device is a real robmqtt client:

from robmqtt import ProductionMQTTClient

self.client = ProductionMQTTClient(
    client_id=f"fleet_{device_id}",
    broker_host=broker_host,
    broker_port=broker_port,
    max_queue_size=500,
    db_path=f"./data/{device_id}.db",
    min_backoff=2,
    max_backoff=30,
    log_dir=f"./logs/{device_id}",
)

self.client.connect()
self.client.start()

Each device publishes three kinds of message, and the QoS and priority differ by importance:

# Telemetry — frequent, can tolerate eviction under pressure
self.client.publish(topic=f"fleet/{id}/telemetry",
                    payload=json.dumps(payload), qos=1, priority=5)

# Status — operational health, higher priority
self.client.publish(topic=f"fleet/{id}/status",
                    payload=json.dumps(payload), qos=1, priority=8)

# Boot/alert events — must not be lost, highest priority, QoS 2
self.client.publish(topic=f"fleet/{id}/events",
                    payload=json.dumps(payload), qos=2, priority=10)

This is the priority system from part 1 in action. If the broker goes down and the offline queue fills, routine telemetry (priority 5) gets evicted before status messages (priority 8), and event messages (priority 10, QoS 2) are protected.

The status messages even report the client's own internal state, pulled straight from robmqtt:

stats = self.client.get_statistics()
payload = {
    "queue_depth":     stats.get("offline_queue_size", 0),
    "inflight_count":  stats.get("inflight_count", 0),
    "is_connected":    stats.get("is_connected", False),
    "reconnect_count": stats.get("reconnect_count", 0),
}

So the simulated fleet reports on its own connectivity health — which means you can build a dashboard that shows the offline queue filling and draining in real time.

Watching the offline queue work

This is the part I find satisfying. Start a device:

python device_simulator.py --device-id device_001 --device-type gateway

[device_001] Started — type=gateway location=warehouse_a

Now kill the broker. The device keeps publishing — but the messages are now being written to SQLite instead of sent:

sudo systemctl stop mosquitto

The device doesn't crash. It doesn't error. It just quietly queues. The queue_depth in the status payload climbs: 5, 12, 28, 45...

Bring the broker back:

sudo systemctl start mosquitto

The queue drains automatically. Every reading that piled up during the outage is delivered, in priority order. The queue_depth falls back to zero. Nothing was lost.

That's the whole point of the library, demonstrated in a way you can watch happen.

Why a simulator is worth building

Even if you have real hardware, a simulator earns its place:

It lets you test at scale you don't have hardware for. You can run 15 simulated devices on your laptop and see how your pipeline, database, and dashboards behave under fleet-level load.

It gives you reproducible failure scenarios. Killing a broker on demand is a lot easier than waiting for a real 4G connection to drop in the field.

It produces clean test data with known properties. You injected the anomalies, so you know exactly what your anomaly detection should catch.

And — as a bonus — it doubles as living documentation for how to use your client library. The simulator is the example.

What's next

In part 3 I'll cover running this at fleet scale — launching many devices at once and feeding their telemetry through an analytics pipeline into a live dashboard.

The full simulator code is on GitHub alongside robmqtt:

PyPI: pypi.org/project/robmqtt
GitHub: github.com/ranaweerasupun/resilient-edge-mqtt-client

How do you test your IoT pipelines — real hardware, simulators, or something else? I'm curious what others do — let me know in the comments.

Why Your MQTT Client Is Silently Losing Messages (And How I Fixed It) - robmqtt

Supun Sriyananda — Tue, 26 May 2026 14:01:19 +0000

Why Your MQTT Client Is Silently Losing Messages (And How I Fixed It)

I learned this the hard way.

I was building a sensor system for a field deployment — Raspberry Pi units publishing temperature and humidity data over 4G cellular to an MQTT broker. The dashboard looked fine. The graphs looked fine. Then one day I compared the raw sensor logs against what actually made it to the broker.

Thousands of readings. Gone. No errors. No warnings. Just gone.

The culprit? paho-mqtt's default behaviour when the broker is unreachable: it silently drops your message and moves on.

After losing enough data I wrote a library to fix it. It's now on PyPI as robmqtt.

pip install robmqtt

But before I show you how it works, let me show you exactly what the problem is — because it's subtler than most people realise.

The Problem with Standard MQTT Clients

When you call client.publish() in paho-mqtt and the broker is unreachable, one of two things happens:

The message is silently discarded (QoS 0)
The message is queued in memory for QoS 1/2 — but that queue is lost on process restart, and there's a second gap that even QoS 1 doesn't close

That second gap is the sneaky one. Here's what happens with QoS 1:

The message was "sent" from your perspective. It was never confirmed from the broker's perspective. And paho has no mechanism to track this gap across reconnections.

On a stable data centre network, this almost never matters. On a Raspberry Pi running on 4G cellular in a field cabinet, it happens constantly.

What a Resilient Edge Client Actually Needs

After losing enough data, I sat down and wrote out what a proper edge MQTT client needs to do:

1. Persist offline messages to disk
If the broker is unreachable when publish() is called, the message should be written to disk and replayed later. Not held in memory — memory is lost on restart.

2. Track in-flight messages separately
Messages that have been handed to the broker but not yet ACK'd need to be tracked. On reconnect, they must be re-sent before any queued messages start draining.

3. Priority-based eviction
When the queue fills up, not all messages are equal. A critical alarm should survive. A routine telemetry reading from 6 hours ago should not block it.

4. Exponential backoff on reconnect
A fleet of 50 devices coming back online after a broker restart should not all hammer the broker at the same second.

5. Thread-safe storage
The MQTT network thread and your application thread are both touching message state. This needs to be safe without forcing the caller to think about locking.

None of this is exotic. All of it is missing from the standard paho-mqtt client when used out of the box.

How robmqtt Solves It

Here's the architecture:

The Offline Queue

When the client detects it's disconnected, publish() routes to an OfflineQueue backed by SQLite:

# Simplified from offline_queue.py
class OfflineQueue:
    def enqueue(self, topic, payload, qos, priority):
        with self._lock:
            self._db.execute("""
                INSERT INTO queue (topic, payload, qos, priority, timestamp)
                VALUES (?, ?, ?, ?, ?)
            """, (topic, payload, qos, priority, time.time()))

    def dequeue_batch(self, batch_size=10):
        with self._lock:
            # Highest priority first, then oldest first within same priority
            return self._db.execute("""
                SELECT id, topic, payload, qos FROM queue
                ORDER BY priority DESC, timestamp ASC
                LIMIT ?
            """, (batch_size,)).fetchall()

SQLite gives you durability without a separate process. It survives power cycles. The threading lock means your application thread and the drain thread never step on each other.

The Inflight Tracker

This closes the gap QoS 1 leaves open:

# Simplified from inflight_tracker.py
class InflightTracker:
    def track(self, mid, topic, payload, qos):
        """Call this when you hand a message to paho."""
        with self._lock:
            self._db.execute("""
                INSERT OR REPLACE INTO inflight (mid, topic, payload, qos)
                VALUES (?, ?, ?, ?)
            """, (mid, topic, payload, qos))

    def acknowledge(self, mid):
        """Call this in on_publish callback."""
        with self._lock:
            self._db.execute("DELETE FROM inflight WHERE mid = ?", (mid,))

    def get_all_pending(self):
        """Call this on reconnect — re-send everything unacknowledged."""
        with self._lock:
            return self._db.execute(
                "SELECT topic, payload, qos FROM inflight"
            ).fetchall()

On reconnect, the client replays all inflight messages first, then starts draining the offline queue. Delivery order is preserved.

Priority Eviction

Each message gets a priority from 1 (lowest) to 10 (highest):

# Routine telemetry — can be evicted when queue is full
client.publish(
    topic="sensors/temperature",
    payload='{"value": 23.5}',
    qos=1,
    priority=3
)

# Critical alert — survives eviction, displaces old telemetry
client.publish(
    topic="alerts/critical",
    payload='{"type": "over_temp", "value": 87.2}',
    qos=2,
    priority=9
)

When the queue hits capacity, the lowest-priority messages are evicted first. Your critical alerts are never blocked by a backlog of stale routine data.

Using robmqtt

Install:

pip install robmqtt

Basic usage — this is everything you need:

from robmqtt import ProductionMQTTClient
import json

client = ProductionMQTTClient(
    client_id="field_device_001",
    broker_host="mqtt.yourdomain.com",
    broker_port=1883,
    max_queue_size=5000,    # holds ~5000 messages during outages
    min_backoff=2,          # start retrying after 2s
    max_backoff=60,         # cap retry interval at 60s
    db_path="./device.db",  # SQLite lives here — survives reboots
)

client.connect()
client.start()

# From here just call publish() — routing is handled internally.
# Connected: sends directly and tracks inflight.
# Disconnected: writes to SQLite, drains automatically on reconnect.

while True:
    reading = read_sensor()
    client.publish(
        topic="sensors/temperature",
        payload=json.dumps(reading),
        qos=1,
        priority=5,
    )
    time.sleep(30)

The application code doesn't need to know whether the broker is reachable. That's the point.

Check what's happening at runtime:

stats = client.get_statistics()
print(stats)
# {
#   'is_connected': True,
#   'offline_queue_size': 0,
#   'inflight_count': 2,
#   'reconnect_count': 4,
#   ...
# }

TLS is supported if your broker requires it:

client = ProductionMQTTClient(
    client_id="secure_device_001",
    broker_host="mqtt.yourdomain.com",
    broker_port=8883,
    use_tls=True,
    ca_certs="/etc/ssl/certs/broker-ca.crt",
    username="device001",
    password="your_password",
)

Seeing It in Action

The repo includes test_13.py, a simulation designed specifically to demo the offline behaviour:

# Terminal 1 — run the simulation (publishes every 5 seconds)
python test_13.py

# Terminal 2 — simulate a network outage
sudo systemctl stop mosquitto

# Watch messages queue up in Terminal 1
# Queue stats print every 10 readings

# Restore connectivity
sudo systemctl start mosquitto

# Watch the offline queue drain automatically — zero messages lost

The queue drain happens in a background daemon thread. Your application code does nothing. It just works.

Real-World Context

I've deployed this pattern on:

Battery management systems — monitoring cell voltages and temperatures in production energy storage systems. A 10-minute broker outage during a network switch should not cause a gap in the battery health record.
Robotics telemetry — ROS2 robots publishing sensor and status data. Process restarts during OTA updates should not lose the last known state.
MQTT edge gateways — aggregating data from downstream sensors over serial or CAN and forwarding to a cloud broker over 4G. The gateway may reconnect dozens of times per day.

In all of these cases, the pattern is the same: treat disconnection as normal, not exceptional. Design the client to buffer, not to fail.

Who This Is For

robmqtt is specifically for edge device deployments where:

Network connectivity is unreliable (cellular, Wi-Fi roaming, VPNs)
Process restarts happen (watchdog resets, power cycles, OTA updates)
Message loss has real consequences (industrial monitoring, remote sensors, fleet telemetry)
You don't want to build and maintain this infrastructure yourself

If you're running MQTT on a stable cloud-to-cloud connection, paho-mqtt alone is probably fine. If you're deploying on Raspberry Pi, industrial gateways, field sensors, or anything running on 4G/LTE — this is for you.

What's Next

The Prometheus metrics endpoint is on the roadmap. The structured logging already writes .jsonl metrics files — exposing them via HTTP is a small step and would make robmqtt slot naturally into standard observability stacks.

If you try it and hit an issue, open a GitHub issue. If you want a feature, open a discussion.

PyPI: pypi.org/project/robmqtt
GitHub: github.com/ranaweerasupun/resilient-edge-mqtt-client

Have you run into MQTT message loss on edge devices? How did you handle it — drop a comment below.