I smashed a bug so hard and came out with a Python library

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.

This is the story of my bug spray.

I build sensor systems that live outdoors. Things like Raspberry Pi units on 4G cellular, embedded gateways, battery monitoring hardware in places nobody visits twice a year. On those networks, connection reliability isn't a simple nice-to-have.

I was a junior back then, and I lost data for weeks before I understood why. When I finally went looking for answers, all I found were scattered fixes. Someone had commented on a years-old thread, "this worked for me". Without any consistency anywhere.

So I patched. Patch after patch, bug after bug, year after year. At some point I thought: why not smash this thing once and for all, for everyone?

And so I did. Smashed, and smashed, and smashed. The final result? An open source python library.

The device that went deaf

Let me tell you about the bug first. So that you know what I smashed and you don't feel pity for it.

Picture a busy restaurant. You're working the pass out front; the kitchen is through the back, and the two of you talk over a radio link. Every thirty seconds the kitchen shouts, "All dishes on time!" Then the food stops coming out. You check the channel, the power light and the microphone. And Everything is perfect. The kitchen keeps cheerfully reporting they're doing great, and ignores every order you scream back.(Sounds annoying, right?)

What happened is that the signal hissed for half a second earlier that night. The link dropped and came back instantly. But the radio system had been set up with a rule: when a link drops, wipe everything about it. The kitchen's standing request to be told about incoming orders was part of that everything. So it went ahead and wiped everything!. The kitchen isn't muted and isn't broken. Its request simply isn't on the list anymore, and it has no way of knowing that. So there are no warining lights or warning beeps beacuse there is no failure to report, since nothing failed.

The system expects the kitchen to re-announce "we're listening on channel four" itself, every single reconnect, forever, unprompted. If it doesn't, it wakes up half alive: still broadcasting to the dining room, deaf to every order coming in. And the only channel it has for telling you something is wrong is the half that still works.

The radio wasn't defective. It did exactly what it had been told to do. And who was the person told it? Me!. I told it months earlier, when I set it up. It was just never designed for a kitchen four hours away with nobody standing next to it.

MQTT is a simple messaging system used by smart gadgets to talk to servers, acting like a two-way radio channel where devices must specifically "subscribe" to a frequency to hear messages. By default, if a device loses its connection even for a second, the system resets and forgets what channel it was listening to, requiring the device to manually tune back in. Paho MQTT is an open-source library that gives you this MQTT messaging protocol.

Once I started looking, it wasn't the only bug. A bug spray is required!

The deaf device showed me a buch of other bugs and issues with this whole thing.

Reconnection is your problem, not the library's.

paho has reconnect_delay_set, but only loop_forever() honours it. With loop_start() — which is what you want the moment the client isn't the main thread — you're writing the retry loop yourself. I wrote the naive one. Fixed a five-second retry, like everybody's first attempt.

Now, having fixed the reconnection bug, I wanted to try it on a device. It worked. I was happy. Wasn't so hard, I thought. So a while later I proudly rolled it out to ten devices.

It died right away. Ten devices, one outage, and every one of them waiting exactly five seconds before knocking again — in perfect unison, forever. The broker came up, took ten simultaneous connections in the same instant, and went straight back down. I waited. It never resurrected itself. It couldn't: each restart just called the stampede back.

Ten devices were enough to demostrate this bug. So, yes. I fell to a bug from a bug. The fix was exponential backoff with a jitter, so they don't re-synchronise. This is the difference between recovery and a self-inflicted DoS.

#Backoff needs a cap and jitter.
# And without the jitter, ten devices that dropped together retry together. That's what killed my broker.
delay = min(1.0 * 2 ** attempt, 300)   # cap: never wait over 5 min
time.sleep(random.uniform(0, delay))   # jitter: don't come back in unison

The bug came back!

Smash a bug hard enough and it comes back as a phoenix. Karma, probably. My new backoff loop retried forever, which is what I wanted, because a device four hours away has to keep trying. But it had no idea why a connection ended. So when I hit Ctrl-C, my shutdown handler closed the connection, my own retry loop saw a dead connection, and it immediately opened a new one. The process wouldn't exit. I had made it unkillable by making it reliable.

There is a simple fix, though. When paho tells you the connection dropped, it hands you a number. Zero means you closed it yourself. Anything else means it broke. Only reconnect on "anything else". Nothing does this for you.

#The bug: any disconnect triggers a reconnect.
def on_disconnect(client, userdata, rc):
    reconnect()   # including the disconnect you asked for

#The fix is one number.
def on_disconnect(client, userdata, rc):
    if rc == 0:
        return        # you closed it. leave it closed.
    schedule_reconnect()

The first connection is a different path, and it usually isn't retried at all.

All that backoff logic I just wrote lives in on_disconnect — the callback that fires when a connection we already had goes away. But a device that boots before its 4G modem is ready never had one. connect() raises, main() dies. Because the modem is not ready to communicate. And the process exits before any of my retry code ever runs. When systemd restarts the service if the modem still isn't up, it dies again. That's not a retry loop I designed; it's just a process crashing over and over.

And the final boss, which is the worst: messages just vanish.

My five-minute ceiling had a consequence I hadn't thought about. After a bad drop, the device deliberately just sits there doing nothing for up to five minutes. Which means every reading it takes during that window has nowhere to go. publish() doesn't complain about this. It hands back a result code saying the message never left. But that is all. Paho can queue messages for you, but that queue lives in memory, so it dies with the process. A power cycle in the field takes your data with it. Gone.

I tried a few of my own patches over the years for this without ever fixing it properly. Sensor data is the entire reason the device is out there. That's when I stopped patching and started writing a queue that saves messages to disk, so they survive a restart.

All these four different failure modes had one thing in common: all of them fail quietly.

So I made the thing I'd been looking for: The bug spray!

Everything above became robmqtt — a resilient MQTT client for edge devices.

• Offline queue in SQLite — unreachable broker means messages get written to disk, not dropped. They survive restarts and power cycles.
• Inflight tracking — sent-but-unacknowledged messages are tracked separately and re-sent on reconnect.
• Priority eviction — when the queue fills, low-priority telemetry goes before critical alerts.
• Exponential backoff — no stampede.
• Subscription registry — subscriptions are restored automatically on every reconnect. No on_connect bookkeeping, no deaf devices. It also means you can call subscribe() before connect(), which is how it should have worked in the first place.
• Structured logging throughout — so the next person doesn't lose days to a problem that leaves no trace.

The application code never has to know whether the broker is reachable. That was the entire goal.

from robmqtt import ProductionMQTTClient
import json, time

client = ProductionMQTTClient(
    client_id="field_device_001",
    broker_host="mqtt.yourdomain.com",
    broker_port=1883,
    max_queue_size=5000,
    db_path="./device.db",
)
client.connect()
client.start()

while True:
    client.publish(
        topic="sensors/temperature",
        payload=json.dumps(read_sensor()),
        qos=1,
        priority=5,
    )
    time.sleep(30)

That's it. The bug spray!

What I'd tell myself at the start

Silence is not success. A green dashboard measures the things you thought to measure. My device was reporting perfect health while it was completely deaf to commands, because I'd never built anything that could notice the difference.

If a library leaves it to you, it will not warn you. Reconnection, re-subscription, retry and queueing in paho isn't broken for leaving these open. It's a protocol client, not a delivery guarantee. But every gap it leaves is a gap that fails in actual production.

Write the log line you'd want at 2am. Tests and structured logging exist in robmqtt for one reason: I spent days chasing something that left no trace, and I refuse to do that to whoever picks this up next.

The thing that cost me weeks is now one pip install away. That's all I wanted.

pip install robmqtt

PyPI: https://pypi.org/project/robmqtt/
GitHub: https://github.com/ranaweerasupun/resilient-edge-mqtt-client

---

Built by Supun Sriyananda — R&D Engineer working on embedded and IoT systems.