BlackBull goes multi-protocol (part 2) — an MQTT 5 broker with no locks and a single owner

#webdev #mqtt #http

Part 1 rebuilt BlackBull's connection dispatcher so BlackBull's core no longer
assumes every connection is HTTP. With that seam in place, a non-HTTP protocol
can register a binding and be served like any other. This post is the first real
consumer of that seam: a full MQTT 5.0 publish/subscribe broker, running in
the same process as your HTTP routes, on the standard 1883 port.

If you haven't crossed paths with MQTT: it's a lightweight
publish/subscribe protocol that powers most IoT sensor-to-cloud communication.
Clients publish messages to topics (like sensors/room1/temperature), and
a broker routes them to every client that subscribed to a matching pattern.
No polling, no request/response pairing — a sensor just publishes and moves on.
It's simple on the outside; building the broker that routes those messages
correctly under concurrency is where it gets interesting.

A broker is the hard case for concurrency. Subscriptions, sessions, retained
messages, and Last-Will templates are all shared mutable state, touched by every
connection at once. The obvious implementation reaches for locks. BlackBull's
reaches for the actor model — and the result needs to own that state safely in
two dimensions: within a worker process, and across several of them.

---

Dimension one: three guarantees inside a worker

The actor model gives the broker three things that a lock-based design would
have to build by hand: shared state without locks, trivial lifetime
management, and backpressure isolation. All three fall out of the same
rule — one actor owns the state, no one else touches it.

No locks on shared state

Inside a single process the broker is three kinds of actor, each in its own
module:

Actor	Count	Owns	Inbox carries
`BrokerActor`	1 per process	all routing / session / retained state	client control events
`MQTT5Actor`	1 per connection	one socket's write side	outbound packets
`TapActor`	1 per process	nothing (stateless dispatch)	published messages for `on\_message` taps

The load-bearing property: BrokerActor processes its inbox serially. Two
PUBLISH packets from two different connections are handled one after another, never
concurrently. So the routing table and the session dictionaries are plain Python
objects — no locks, no asyncio.Lock, no shared mutable state reached from two
tasks at once. That is the core reason to use the actor model here: serial
inbox processing turns "shared mutable state" into "state owned by exactly one
actor," and the need for locks evaporates.

**Actor model vs locks — when does each win?**

Scenario Actor (serial inbox) Lock (asyncio.Lock)

Many writers, one shared state **Wins.** Messages queue; throughput stays flat regardless of writer count. Contention grows with writers; throughput degrades.

Few writers, mostly reads Message dispatch overhead per access — paid even when uncontested. **Wins.** Fast acquire/release; reads often lock-free with asyncio patterns.

Parallelism within the state owner Can't. One actor = one task = one core. **Wins.** Fine-grained locking lets multiple cores work on different parts.

Debugging Harder. Bugs span actor boundaries; stack traces don't cross send() calls. **Wins.** PYTHONASYNCIODEBUG=1 surfaces slow locks; deadlock detection built in.

Deadlock risk Low, but not zero. Circular send() between two actors each awaiting the other's reply can deadlock. **Real risk.** Two locks acquired in different order → classic deadlock.

Error isolation **Wins.** A crashed actor doesn't corrupt its peers' state. One task holding a lock that crashes can poison the lock (needs try/finally discipline).

A broker hits the first row on every connection: many writers converging on one
routing table. That's why the actor model fits here — not because it's always
better, but because the access pattern aligns with what actors are good at.
For a different workload (say, a connection pool with mostly reads), a lock
may well be the simpler, faster choice.

Scenario	Actor (serial inbox)	Lock (`asyncio.Lock`)
Many writers, one shared state	Wins. Messages queue; throughput stays flat regardless of writer count.	Contention grows with writers; throughput degrades.
Few writers, mostly reads	Message dispatch overhead per access — paid even when uncontested.	Wins. Fast acquire/release; reads often lock-free with `asyncio` patterns.
Parallelism within the state owner	Can't. One actor = one task = one core.	Wins. Fine-grained locking lets multiple cores work on different parts.
Debugging	Harder. Bugs span actor boundaries; stack traces don't cross `send()` calls.	Wins. `PYTHONASYNCIODEBUG=1` surfaces slow locks; deadlock detection built in.
Deadlock risk	Low, but not zero. Circular `send()` between two actors each awaiting the other's reply can deadlock.	Real risk. Two locks acquired in different order → classic deadlock.
Error isolation	Wins. A crashed actor doesn't corrupt its peers' state.	One task holding a lock that crashes can poison the lock (needs `try/finally` discipline).

MQTT5Actor is the sole writer to its socket. Its run() loop drains an
inbox of outbound packets; a sibling reader task decodes the wire and sends
control messages to the BrokerActor. Even replies the connection could answer by
itself — PINGRESP, AUTH — are routed back through its own inbox, so run()
stays the only thing that ever touches the socket's write side. No cross-task
write races, by construction.

Why Last-Will delivery is free with actors

MQTT's Last-Will-and-Testament is a message the broker delivers on behalf of a
client that disconnects uncleanly — "if I disappear, tell everyone X." The
hitch: by the time the broker notices the disconnect, the client's connection
actor is already tearing itself down. Who owns the Will at that moment?

Because BrokerActor outlives every connection, the answer is straightforward:
the dying MQTT5Actor hands the Will template to BrokerActor during
teardown, and BrokerActor routes it to live subscribers. No global registry
kept alive just for cleanup, no cross-actor lifetime coupling — the long-lived
broker makes Will delivery a trivial handoff between two actors that already
exist.

Slow observers can't stall the broker

@mqtt.on\_message handlers — application taps that observe broker traffic —
run on a separate plane via TapActor, which consumes a bounded inbox. The
connection offers each published message without blocking and returns
immediately. If taps fall behind, the newest message is dropped and a running
dropped-count is logged (silent loss is the one unacceptable outcome, so the
count is always surfaced). A slow tap back-pressures nothing.

Here is a controlled side-by-side — one build, one machine, one session, with the
dispatch mode (inline vs actor) as the only variable, tap queue depth 256:

tap delay	inline throughput	actor throughput	inline p99	actor p99
0	61,063/s	52,529/s	5.64 ms	6.59 ms
1 ms	881/s	52,835/s	448.66 ms	6.31 ms
5 ms	193/s	59,817/s	2,047.97 ms	5.81 ms
25 ms	39/s	59,500/s	10,135.94 ms	5.82 ms

Inline delivery collapses as 1/delay with p99 climbing past ten seconds; actor
delivery stays flat — ~52–60k msg/s independent of tap speed. The trade is
explicit: stable delivery bought with bounded tap coverage (and a logged drop
count under overload).

A historical note worth its own line: MQTT is the first production code in
BlackBull that uses the actor inbox for real. The HTTP actors override run()
and call each other through direct method calls — the inbox was defined but
latent on that path. The broker is what proves the inbox works at scale (451
conformance tests, all green).

---

Dimension two: a single owner across workers

Serial inboxes solve concurrency inside one process. But BlackBull runs HTTP
across multiple worker processes for throughput — and that collides head-on with
a broker.

The MQTT 5 spec assumes one logical broker: a client's session, its queued
messages, and its subscriptions are one coherent piece of state. Shard the
broker across four worker processes with no shared store and a client that
reconnects to a different worker finds its session gone. So the broker must
have a single owner.

The naive resolution is "any stateful protocol forces workers=1 for the whole
process" — which throws away HTTP's multi-worker scaling to satisfy MQTT. v0.44.1
does something better:

The master process binds the broker's :1883 socket once and hands it to worker 0 only. The broker lives there, single-owner, exactly as the spec wants.
HTTP is stateless, so it scales on every worker. app.run(port=8000, workers=4) alongside MQTTExtension(port=1883) now runs HTTP on all four workers while MQTT runs on worker 0.
If worker 0 crashes, the master respawns it and it re-inherits the still-open listener — the broker's single-owner identity is tied to "worker 0," not to a particular PID.
BB\_SOCKET\_REUSEPORT=1 gives each HTTP worker its own kernel accept queue for the best load distribution; the broker port is always bound without SO\_REUSEPORT, by design — a single owner must not have its socket load-balanced away.

(Auto-reload (--reload) still pins workers=1 — the exec-based socket handoff
doesn't yet carry protocol listeners. A known edge, documented.)

So the two dimensions compose: within a worker, the actor model guarantees
lock-free state, trivial lifetimes, and backpressure isolation; across
workers, single-owner affinity means the broker stays coherent while HTTP
scales past it.

---

What it cost the HTTP path

Nothing. Adding the broker is a self-contained extension on the part-1 seam — the
core BlackBull class carries zero MQTT-specific code. The same release that
shipped the broker measured zero regression on the HTTP/1.1 hot path. A second
protocol arrived and the first one didn't notice.

A closing thought. BlackBull already has a second extension on PyPI —
blackbull-htcpcp, an
implementation of HTCPCP (RFC 2324), the HTTP extension for controlling
coffee pots. Now imagine an MQTT-enabled coffee machine: the HTCPCP extension
could serve BREW /pot on :80 while the broker receives brew-complete
notifications on :1883, sharing the pot's state in one Python dict. Two
RFCs, one process, no sidecar. The MQTT broker is just the first proof that
the part-1 seam delivers on that promise.

Next, part 3: what all this looks like from an application author's chair — one
app.py, two protocols, and machine-readable docs for both.

---

Source: github.com/TOKUJI/BlackBull
Docs: tokuji.github.io/BlackBull