DEV Community: Prajwal Mahajan

Building toykv: a from-scratch persistent KV in Go, and why I took the opposite call from toymq three times

Prajwal Mahajan — Wed, 17 Jun 2026 16:39:53 +0000

A retrospective on toykv — a single-node, in-memory key-value store with append-only-file persistence, RESP2-compatible on the wire, three binaries, and a v1.0.0 I shipped after about four weeks of work.

TL;DR — and the rule I want you to remember

Some rules generalise across systems by inversion. toymq taught me zero values are not sentinels. toykv taught me durations are not deadlines. Same shape of bug, opposite axis — and along the way I ended up answering three load-bearing questions in the exact opposite direction from toymq:

I shipped toymq with no version byte on its WAL. toykv's AOF starts with one.
I gave toymq its own wire protocol and a separate WAL record frame. In toykv I write the same RESP arrays I speak on the wire straight into the AOF.
toymq had its own delivery semantics to defend. For toykv I chose RESP2 specifically so redis-cli and go-redis/v9 become free third-party test harnesses.

The proof those opposite answers were right is the crash matrix — nine rows, each pinned by a test file that exists today, each test owned by the part of the system that introduced the risk. Every row is cheap because of one of those three decisions.

The hero rule of toykv — sibling of toymq's lastAcked uint64 == 0 zero-value-sentinel bug, reached from the other direction — is one line:

Absolute deadlines, never relative durations.

Store EX 5 on the wire; write PEXPIREAT 1718659200000 to disk. toymq taught me that zero values are not sentinels. toykv taught me that durations are not deadlines. Same shape of bug, opposite axis.

The numbers: ~5k LOC of Go, 6 ADRs (I budgeted four), 15 journal entries, three binaries (toykv, toykv-cli, toykv-tui), v1.0.0 tagged on 2026-06-17.

What toykv is, and isn't

A learning artifact. Single-node, in-memory, no auth, no TLS, strings only. 18 commands from the Redis surface — GET, SET (with NX/XX/EX/PX/EXAT/PXAT), DEL, INCR/DECR, KEYS, FLUSHDB, EXPIRE/TTL/PEXPIRE/PEXPIREAT/PERSIST, BGREWRITEAOF, and a handful of metadata commands. Under appendfsync=always (the default), throughput tops out in the low thousands of SETs per second on commodity NVMe — a per-write fsync is the durability commit point and the cost is honest. If you need a real KV store, reach for Redis or Valkey. If you want to understand what "durable" means in one, build one.

The crash matrix

I lay it out the way I read it: every row of fault surface has exactly one test file, and the test lives next to the layer that introduced the risk. Composed-fault tests sit on their own at the bottom because composing faults is a different job from proving any one of them.

Fault surface	Risk	Test file	Invariant proven
AOF append + replay	acked SET / DEL lost on SIGKILL under `fsync=always`	`internal/server/aof_crash_test.go`	Every record acknowledged before the kill replays on restart
Partial-tail handling	crash mid-record corrupts replay state	`internal/aof/replayer_test.go`	Replay rejects a torn record; offset reported, server refuses to serve until truncated
TTL lock-upgrade race	sweeper drops an unexpired key under load	`internal/store/sweeper_test.go`	N writers × 1 Hz sweeper → zero spurious `(nil)` for unexpired keys
TTL crash round-trip	TTL records lost across SIGKILL + restart	`internal/server/aof_ttl_crash_test.go`	v2 replay accepts v1 records; expiry decodes round-trip
`BGREWRITEAOF` during writes	rewrite races a concurrent SET; tail loses data	`internal/aof/rewriter_test.go`	Side-buffer captures all live appends; rewrite + tail merge with no loss
Crash during rewrite	partial `.aof.tmp` left, no canonical AOF survives	`internal/server/aof_rewrite_crash_test.go`	Exactly one of `{old, new}` present after restart; replay consistent
Shipped binary protocol drift	`redis-cli` / `go-redis` regression invisible to in-process tests	`test/e2e/protocol_*_test.go`	Byte-compat for every shipped command
`BGREWRITEAOF` + restart	restart after compaction loses data	`test/e2e/rewrite_restart_test.go`	Mixed-workload state survives rewrite + restart
Composed faults (kill + pause + rewrite + writes)	failure modes only visible when multiple faults overlap	`test/chaos/invariants_test.go`	Acked-SET survival, monotonic INCR, no panic across soak

The protocol-compatibility layer (rows 7–8) deliberately introduces zero new crash invariants. It verifies that the bytes leaving the socket match the spec third-party clients expect; everything below the socket has already been proven elsewhere. The chaos suite (row 9) is the only layer that composes faults. Composed-fault tests as the primary durability proof are slow and flaky; composed-fault tests as a release-confidence soak after each component is independently proven are exactly what you want.

The thesis I want you to take away: every row is cheap because of one of the three decisions below. Where a crash matrix becomes expensive — where the test file is 800 lines, or two rows have to share state — that's usually the symptom of an architecture decision made elsewhere.

The first three layers introduce new fault surface, each proven in isolation by a dedicated crash test. The protocol layer verifies the shipped binary speaks the spec; it adds no new crash invariants. The chaos layer composes everything underneath into one soak. That shape is the proof the architecture is paying its rent.

Inversion 1 — a version byte you'll thank yourself for, and absolute deadlines

toymq's WAL has no version byte. The argument I wrote in that retrospective is good: defending the wrong byte forever is worse than the one-line migration of adding one later.

toykv's AOF starts with eight bytes: "TOYKV\x00\x00" plus a one-byte version. The reasoning is also good, for a different shape of problem: the AOF outlives the binary that wrote it. If I run toykv on a tempdir on Monday and the same dir with a newer binary on Friday, the Friday binary has to know what assumption the Monday bytes were committed under. There is no person to ask. The byte is the answer.

I made the call before I wrote a line of replay code, and wrote it up as an ADR alongside the first AOF commit. It paid for itself inside four weeks: I needed a new on-disk shape for TTL, and the version byte let v2 binaries accept v1 files transparently. The replay loop is the same eight-line dispatch table; v2 just knows about a PEXPIREAT token v1 didn't.

The rule: the cheaper migration is the one your future self can opt into, not the one their replay code is forced into. A version byte you don't use is a no-op. A version byte you didn't write is a migration script.

Absolute deadlines, never relative durations

The companion ADR made the second call: every TTL command I accept on the wire rewrites to absolute PEXPIREAT <unix-ms> before it touches the disk. A wire SET k v EX 5 becomes an AOF SET k v + PEXPIREAT k <now-ms + 5000>. The wire stays ergonomic; the disk stays unambiguous.

Imagine the alternative. Store EX 5 verbatim in the AOF. Crash. Restart twelve hours later. Replay reads EX 5 and applies it against time.Now() — every TTL in the file just got extended by twelve hours. Worse, the bug is silent: keys don't disappear, they linger. The test that catches this is a multi-day fixture, because the only way to notice is to wait past the original expiry. That's the toykv-shaped sibling of toymq's lastAcked == 0 sentinel collision — a value that looks valid until the system restarts in a state it didn't anticipate. Same shape of bug, different axis.

The rule: absolute deadlines, never relative durations. The wire can speak in human-friendly relative time; the disk has to speak in moments. A duration without a reference point isn't a deadline — it's a wish.

This is the rule I earned. Everything else in the post is the architecture I built to make the rule cheap to enforce.

Replay reads PEXPIREAT k 1718659200000. The math works regardless of when replay runs — a year later, the answer is "this key was already expired"; a second after the crash, "this key has 4.999 seconds left." There is no clock to argue with. The wire stays ergonomic; the disk stays unambiguous.

The AOF on disk

The file format is deliberately boring:

data/
└── appendonly.aof    ← header + RESP-encoded record stream

One file. No manifest, no index, no segment rotation in v1. The header is eight bytes; everything after is a stream of RESP arrays. The recovery scan walks the file from offset zero on Open — O(disk), but correct by construction. A manifest would be a second consistency problem on top of the first.

Field	Type	Size	Notes
`magic`	bytes	7	Literal `"TOYKV\x00\x00"` — fails fast on a wrong file
`version`	`u8`	1	`0x01` (strings only) or `0x02` (adds `PEXPIREAT` for TTL)
`record_*`	RESP	variable	Same `*<n>\r\n$<len>\r\n…` frame the wire speaks

There is no per-record CRC. RESP framing already catches truncated bulks and mismatched array counts; adding a CRC for a learning artefact is belt-and-braces I deliberately deferred. The first real-world corruption that slips past RESP-level parsing is the moment to add one — not before.

Inversion 2 — write the bytes you'd send

The journal entry that wrote itself, docs/journal/02-aof.md, has this line:

"The whole AOF format is RESP arrays on disk. Append calls the existing resp.Writer against a buffered file handle; Replay calls the existing resp.Reader. One codec, two consumers."

That's the architectural claim. Everything I did downstream of it is mechanical. The replay path is nine lines of Go: skip the header, read a RESP array, dispatch it, repeat to EOF. During replay s.aof == nil, so the appendIfLive call inside each mutating handler is a silent no-op. The same nine lines handle real traffic and crash recovery.

The AOF-replay invariant — every record acknowledged before the kill replays on restart — collapses to a single ordering question: does the +OK cross the network before or after file.Sync() returns? Under appendfsync=always, the answer is after. The proof is operational: the crash test (internal/server/aof_crash_test.go) writes ~90 SETs against a self-re-exec child server, sends SIGKILL mid-stream, restarts, and checks every +OK against the post-replay state. -count=10 after the first pass found zero flakes.

I drew this diagram in the same shape as the WAL diagram in the toymq retrospective on purpose. A reader who has seen both will notice that the durability ordering is identical — that's the thing the two projects agree on. The codec choice is what's different, and the codec choice is what made the replay path nine lines instead of a hundred.

The rule: write the bytes you'd send. One codec is cheaper than two consistent codecs, and the second codec is where the corruption story you don't want lives.

Inversion 3 — RESP because the harness is free

For toymq I invented a protocol. The defence is sound: the protocol is the wire, the wire is the contract, an invented wire is the moment a project decides it can defend its own boundaries.

For toykv I chose RESP2. RESP2 is a testing strategy disguised as a feature. The reason is the protocol-compatibility layer of the crash matrix: integration tests against the shipped binary using third-party clients — redis-cli for byte-level conformance, go-redis/v9 for the API a real Go consumer would see. That layer shipped without a line of client code I wrote. The clients ship for free, by people who have never heard of toykv. They don't care about the implementation; they care about the wire. That's exactly the property I wanted.

The crash-matrix tie-in is the load-bearing one: the protocol layer owns zero new crash invariants. It owns protocol invariants — TTL -2/-1 sentinels, EXPIRE against a missing key returning 0, the byte-exact framing of *3\r\n$3\r\nSET\r\n… — but every command was already covered by a crash test in the layer that introduced it. The protocol layer is the validation that the bytes leaving the socket match the spec the third-party clients expect; everything below the socket has already been proven.

The rule: protocol compatibility is a testing strategy disguised as a feature. Two independent implementations of the wire shake out the spec. No single-author project has that property by default — you have to inherit it deliberately.

Honourable mention 1 — the dual-write `bufio` bug

The interesting bug of the project was not the rename-plus-dir-fsync-plus-fd-swap dance for BGREWRITEAOF that I'd been bracing for. The invariants I'd written up in the rewrite ADR worked the first time. What broke was the dual-write Append path inside the rewriter: sideBuf.Len() == 0 after what looked like a successful write.

Root cause: resp.Writer wraps its target in a bufio.Writer. The live path constructs resp.NewWriter(outerBufio), and bufio.NewWriter(outerBufio) is smart enough to short-circuit — it returns the existing *bufio.Writer rather than wrapping it again. One buffer, one flush. When Rewriter.BeginRewrite constructed a fresh mirror writer against a *bytes.Buffer, the argument was no longer a *bufio.Writer. So bufio.NewWriter(&scratch) created a brand new inner bufio layer. Two buffers, two flush points. Records were "written" but pinned in an inner bufio nothing else was going to drain before DrainAndSwap ran. Two-line fix in 736bf63.

The rule: any write path with two sinks has two flush points; both must be ordered against the swap. A bufio layer is invisible — the same line of Go behaves differently depending on the static type of its argument. Never construct a fresh wrapper of a swappable type against a non-bufio target without immediately scheduling its flush.

The invariant the rewrite ADR commits to is "the canonical file is durable and consistent at every instant until the rename." A silently-empty side buffer would have meant the rename swapped in a file missing the appends issued during the snapshot. The replay test was the green-light proof that both sides of the dual-write are actually dual-writing.

Honourable mention 2 — the test was wrong, not the code

The TTL sweeper is a 1 Hz goroutine that samples 20 keys under a read lock, then upgrades to a write lock for the eviction pass — sync.RWMutex doesn't support upgrade-in-place, so the upgrade is release-and-reacquire-and-double-check. The race test (internal/store/sweeper_test.go, commit bab6033) failed on its first stress run with four violations in 468k reads.

The bug was in the test, not the store. The test captured t0 := time.Now() before Get, but Get's internal time check happens later — sometimes much later if the scheduler preempts. A Get that correctly returned (nil) because the entry's TTL had elapsed by the time Get internally checked would be flagged as "spurious nil for unexpired key." The fix forced the memory model onto paper: I now capture tAfter after Get (upper-bounds the internal check) and atomic.Load(lastExpireAt) before Get (synchronizes-with the writer's atomic.Store). Then tAfter < loadedExpireAt is a real violation; no false positives possible.

The rule: if a sweeper and a writer can race, the test has to schedule them, not hope. A test whose pass/fail depends on the scheduler's whim is a test that finds the race detector's bug, not your code's.

ADRs as crystallization — and the ADR I almost didn't write

I budgeted four ADRs for v1.0.0 and shipped six. The interesting one is the TUI ADR.

The TUI ADR (Bubble Tea with an injectable Doer) is where the budget actively pushed against the code. The TUI was the first part of the project that took a third-party dep beyond go-redis/v9 in tests, and the choice — bubbletea vs tview vs hand-rolled — was non-trivial. I knew it was non-trivial because every two days, in a different context, I caught myself re-deriving the same argument for the same choice. That's the signal: when the same reasoning costs me a second time, I didn't decide; I waited.

I had two options. Force the choice into one of the existing four ADRs and write a worse ADR. Or break the budget and write the TUI ADR on its own. I broke the budget. The post-mortem is that the budget itself was the wrong abstraction. I had treated ADR count as a proxy for noise, when the actual signal was ADR quality — and a forced-fit ADR is noisier than two clean ones.

The discipline, identical to the one I wrote in the toymq post: write the ADR the moment the decision is forced by code, never before. ADRs written ahead of code force the code to fit the ADR. ADRs written at crystallization record what the code already decided. The reason that discipline matters is that the TUI ADR was a decision I'd already made three times by the day I wrote it down — I just hadn't admitted I'd made it. The ADR is the admission.

The rule: budget the quality bar, not the count. The cost of an ADR I write is one afternoon. The cost of an ADR I didn't write is a future me re-deriving the same argument in three different files. Those costs aren't comparable, and counting ADRs treats them as if they are.

What I'd do differently

Three regrets from the critical path:

Chaos composition should have shipped with the AOF, not at the end. The test/chaos/ soak (kill + pause + rewrite + writes overlapped) felt expensive enough to defer. It wasn't. Two-thirds of the harness reused the self-re-exec pattern I already had for the AOF crash test. Running a composition soak from the start would have caught the dual-write bug at the point the rewriter landed, because the rewriter would have been driven by a scaffolded integration test instead of an isolated unit test.
Store.Snapshot() should have shipped with the AOF, not landed as a refactor PR when the rewriter needed it. Snapshot is a contract boundary — "give me the current state, expired entries already evicted, in a form I can serialize." I deferred it because the AOF didn't use it yet. The refactor PR that extracted it later was harder to review than the BGREWRITEAOF work itself, because it touched nine files mechanically. Shipping the boundary up front would have cost one extra journal entry.
The ADR budget number was the wrong abstraction. See above. Budget the bar, not the count.

What's next

toykv-go — the SDK extraction. The strategy I committed to is: hold the SDK split until the wire format is locked, then extract internal/client, respfmt, and cmdparse into a sibling repo. v1.0.0 of toykv-go tagged the same day as toykv v1.0.0.

toymessenger — the consumer. The reason toykv and toymq exist as separate projects is that there is a third project, an E2E-encrypted chat, that uses both: KV for session state and presence, MQ for the message log. toymessenger is where the two projects' opposite decisions get tested against a single user.

A toykv v2 only if real use justifies it. The roadmap lists candidate features (lists/sets/hashes, AUTH + TLS, RDB snapshots, observability with INFO + /metrics, SCAN, RENAME), but the rule is the same one Redis uses: don't add a feature until somebody is asking for it and willing to live with the trade-off. A single-node KV that survives a v1 cycle without a v2 ask is a single-node KV that earned the cut.

The one-line version

Write the bytes you'd send. Stamp them with a version. The next person to read them is you.

Source: prajwalmahajan101/toykv. Deeper docs: docs/HLD.md, docs/LLD.md, docs/TESTING.md, docs/BENCHMARKS.md. ADRs: docs/adr/. Release notes: docs/release-notes/v1.0.0.md. Corrections welcome — open a discussion or file an issue.

Companion post: Building toymq — the sibling project this one is in dialogue with.

Building resilience-kit: A Python Resilience Kernel Forged in Production

Prajwal Mahajan — Thu, 11 Jun 2026 08:18:08 +0000

TL;DR

resilience-kit is a framework-agnostic, async-first resilience kernel for Python. Circuit breakers, retries with jitter, token-bucket throttles, SSRF guards with DNS pinning, fire-and-forget audit logs, field-level encryption. Pluggable backends (memory, Redis/Valkey, pybreaker). Thin adapters for FastAPI and Django that wire the primitives into each framework's lifecycle and contain zero business logic. One core, 11 ADRs, two adapters.

The interesting part isn't the primitives — most of them are well-trodden ground. The interesting part is the release rule: v0.1.0 only shipped because I had to upgrade two unrelated starter repos to it and score ≥ 8/10 on each migration. The reports got filed. FastAPI scored 8/10. Django scored 9/10. The gate held. Anything below 8 would have blocked the cut.

That rule — "the kit ships iff the kit survives its own migration" — is what shaped the public surface. Every helper in v0.1.0 (bind_to, from_exception, legacy_env_alias, verify_envelope_contract) exists because the first dogfooding round flagged its absence as a blocker. Every doc gap got patched because the migration reports cited the line number it should have been on. This post is how the kit got built, the five axioms underneath it, the gate that gated it, and the bugs it found in itself along the way.

95 source modules, 52 test modules across unit/contract/integration suites, 11 ADRs.

Five Axioms

I didn't draft these up front. I extracted them after the fact by reading the ADRs and asking what did I refuse to compromise on? They're the load-bearing constraints; everything else follows.

1. One package, many extras. Not a forest of micro-packages.

pip install resilience-kit gives you the core. pip install resilience-kit[redis,pybreaker,fastapi] adds the backends and adapter. No micro-libraries until the surface justifies it. One repo, one CHANGELOG, one tag. Modularity is enforced by import discipline (import-linter with layered contracts) and by entry-point provider discovery, not by package boundaries.

Arrows are hard imports (from resilience_kit.X import Y). Dotted lines are entry-point conformance — no import required, just a Protocol-shaped class registered in the extra's pyproject.toml under [project.entry-points."resilience_kit.cache"] (and siblings). The kit's import graph is acyclic by import-linter contract; consumer apps depend only on the adapter layer.

The bet: a single distributable up to ~10k LOC is simpler to release, version, and depend on than a federation. If the surface ever grows past that, splitting into namespace packages later is mechanical and doesn't break the public API.

2. Protocols, not ABCs.

Every swappable interface — cache, breaker, throttle, audit backend, metrics sink — is a typing.Protocol, not an abc.ABC. Third-party backends import zero kit packages at definition time. They expose a class whose shape matches; mypy --strict verifies the structural conformance.

The rule: your code should conform to a shape, not inherit from a library. This is true dependency inversion — the kit doesn't even appear in the import graph of the thing implementing its protocol. Entry points provide the precedence chain at runtime: explicit > env > entry-point > default.

3. Outer breaker, inner retry. Composition order is correctness.

If you compose a circuit breaker and a retry, the breaker must wrap the retry. If you accidentally invert the stack — retry around breaker — the retry loop will hammer an OPEN breaker, defeating the point of the breaker. This isn't a style preference; it's a correctness property.

I learned this the hard way early on. A caller passed retry_on=(Exception,) to the retry decorator. The decorator happily caught ServiceUnavailableError — the exception the breaker raises to say "stop calling me." The fix (commit f6cebdd) made the dangerous composition fail safe by stripping ServiceUnavailableError from the retry's catch list, no matter what the caller passes:

def _filter_retry_on(exceptions):
    """Drop ServiceUnavailableError from a retry_on tuple."""
    return tuple(
        e for e in exceptions
        if not (isinstance(e, type) and issubclass(e, ServiceUnavailableError))
    )

The rule: the dangerous composition must be the one that fails safe. An inverted stack now fails instantly, because the retry loop refuses to catch the one error a breaker is designed to throw.

4. Async-first with explicit sync bridges. Never trust auto-bridging.

Every primitive's async API is primary; sync wrappers exist only where the primitive is idiomatically sync (Django middleware, DRF throttles, management commands). Crucially, the kit does not use asgiref.sync.async_to_sync. That function caches a thread-local event loop to "avoid overhead" — which means it's a hidden global namespace, and any other library that also caches a thread-local loop will collide with it.

Instead, the kit owns both crossings explicitly:

Long-lived background work (recovery monitor, audit dispatcher): the Django adapter spawns a single daemon thread with its own private event loop. The loop drives the monitor and owns the audit queue. atexit drains the queue on the same loop before closing it.
Short-lived sync→async calls (DRF throttles checking Redis): a per-call asyncio.run(). Sub-millisecond overhead. No shared state. No collision risk.

The rule: a bridge between two worlds must own its own crossing. Don't borrow infrastructure from either side.

5. Adapters are dumb wiring. If your adapter has business logic, your primitive is wrong.

resilience_kit.adapters.fastapi and resilience_kit.adapters.django exist solely to wire kit primitives into each framework's lifecycle: ASGI middleware, FastAPI dependencies, Django AppConfig.ready(), DRF throttle classes, management commands. They translate framework-shaped concepts (a Django MIDDLEWARE tuple entry, a FastAPI Depends()) into kit calls. Nothing else.

If an adapter file ever grows past ~300 LOC, the primitive itself is wrong, not the adapter. This is enforced socially (PR review) and architecturally (import-linter prevents adapters from importing each other or from owning state).

The closing line of this post is the punchier statement of this axiom.

The Dogfooding Gate

The most consequential decision in this project wasn't a code decision. It was a process decision:

"v0.1.0 is a clean cut iff both boilerplate reports score ≥ 8/10."
— docs/RELEASE-PLAN.md §4 verification

I write FastAPI and Django services. Before the kit existed, both starter repos owned the same resilience layer twice — circuit breakers, retries, throttles, SSRF guards, audit logs — copy-pasted and lightly diverged. The kit is the deduplication. But deduplication is easy to talk about and hard to validate; "the kit can replace the boilerplate code" is a claim, not a proof.

So I made it a release gate. Cut a release candidate. Upgrade both boilerplates against it. File a structured report on each migration — blockers, helpers used, missing surface, pain points, doc gaps, ROADMAP suggestions — and a 1–10 outcome score. If either score is < 8, the kit isn't done. Iterate. Re-test.

That cycle ran from 0.1.0rc1 → 0.1.0. Both reports passed:

Repo	Score	Date
`fastapi_boilerplate`	8 / 10	2026-06-10
`django_boilerplate`	9 / 10	2026-06-11

Both reports applied all four primary helpers that shipped in v0.1.0: bind_to(target) for ContextVar mirroring, from_exception(exc) for the kit-shape → adopter-envelope projection, legacy_env_alias() for pre-kit env-var translation, and verify_envelope_contract() for an adopter-side test that the bridge maps every kit exception cleanly. The fact that both reports used the same four — first try, no fallbacks — was the convergent signal that the helper surface was right.

What's interesting isn't that the gate passed. It's what the gate produced.

What the Migration Reports Found

The reports were structured by design — same template, same sections — so the findings could be aligned and compared. Four findings showed up in both:

#	Cross-cutting finding	Target
X1	The public exception bridge lives at `resilience_kit.adapters._envelope.from_exception` — the leading underscore reads as private. Re-export it without the underscore, or rename the module.	v0.1.1
X2	`verify_envelope_contract` raises a flat `AssertionError`. Pytest only surfaces the first failure; programmatic CI dashboards can't introspect a structured result. Return an `EnvelopeContractResult` (list of `(exc_class, ok, reason)`).	v0.1.1
X3	`legacy_env_alias(aliases=...)` replaces the default table. Adopters who want to extend with project-specific aliases must copy the default dict + add. Add `extra_aliases=` that merges, or document the copy-pattern explicitly.	v0.1.1
X4	The rc1 → v0.1.0 migration guide had four documented gaps — different recipes needed Django-specific snippets, ordering was load-bearing in one place, the projection shape was wrong for a required-`code` envelope, and a `request_id=None` patch needed an explicit "top up from your own ContextVar" note.	v0.1.0 doc patch

Two of those (X1, X2) are surface-quality nits — the kit works, but the ergonomics chafed. Two (X3, X4) are real correctness traps — silent override of the default alias table, and migration recipes that worked for FastAPI but not for DRF. Both reports independently flagged all four. That kind of convergence is exactly what dogfooding is for: a single user can rationalize around a sharp edge; two users hitting the same edge means the edge is the bug.

The single-repo findings were sharper.

FastAPI-specific: from_exception(envelope_cls=...) projects the kit's exception details into [{field, message}] per error-list-item. FastAPI's ErrorDetail schema also requires code — so the projection raised a ValidationError at handler time. Cost to diagnose: ~10 minutes. Workaround: drop envelope_cls from the call, translate manually in the handler (six lines). Tracked as a v0.1.1 patch (add code=exc.error_code to the projected items, or add a per-entry callback hook).

Django-specific: the migration guide's verify_envelope_contract example called handler(...).body — that's the FastAPI shape. DRF stores the body at .data, not .body. Doc-only fix; it landed in the same PR as the report.

Smoke proof: the Django report didn't just say the bridge worked. It quoted the response:

GET /api/accounts/me/  (anonymous, 401)

X-Request-Id: 1e43827d5faa4fcfa3551acfcf9caa27
{"request_id":"1e43827d5faa4fcfa3551acfcf9caa27", …}

The same UUID in the response header and in the JSON envelope's request_id field. rc1 returned "request_id":null. v0.1.0 returned the bound value. The bridge — bind_to(request_id_ctx) in a thin BindRequestIdMiddleware slotted immediately after the kit's RequestIdMiddleware — flowed kit → bridge → DRF handler → response. End to end. That's the kind of proof migration reports should ship.

Convergent v0.2 wishlist. Both reports independently asked for the same five things: DjangoSettingsSource (read settings.RESILIENCE instead of forcing env vars), resilience_kit.utils.* (five small modules every boilerplate re-implements: log_sanitization, network, timing, function_logger, data), a GlobalThrottle (process-wide cap; nginx covers it in prod but laptop dev wants the in-process belt), a free-function metrics.record_* shim over MetricsSink to keep adopters' bounded-cardinality guards in place, and multi-alias Redis topology so cache, throttle, and breaker can point at separate instances. That convergence is the v0.2 design spec, half-written for me.

Bugs the Kit Found in Itself

The dogfooding gate is the systemic story. The bugs are the craft story.

`f6cebdd` — outer breaker, inner retry

The bug behind axiom 3. A caller passed retry_on=(Exception,) to the retry decorator. The decorator caught ServiceUnavailableError — exactly the wrong exception. The fix is the _filter_retry_on snippet above. The rule it earned is in axiom 3.

The inverted stack is the one a fresh reader writes first — composition order looks like a style preference until the breaker is OPEN. The _filter_retry_on invariant makes the wrong order fail safely instead of catastrophically: the retry decorator refuses to swallow the breaker's "stop" signal, even when the caller explicitly tells it to.

`bc499da` — `asyncio.Event` rebinding on every restart

RecoveryMonitor held an asyncio.Event created in __init__ and reused across every start()/stop() cycle. asyncio.Event doesn't bind to a loop at construction — it binds lazily, on the first .wait() or .set(). So the first start captured the loop. The second start, on a fresh loop, raised:

RuntimeError: <asyncio.locks.Event object at …> is bound to a different event loop

It surfaced in pytest-asyncio first — two FastAPI integration tests in one session, per-function loop scope. The second test's lifespan opened a fresh loop and called monitor.start(); the Event still held the first test's closed loop. Same shape of bug as Django's runserver autoreload, Gunicorn graceful restart, any ASGI lifespan that opens twice. The fix:

# In src/resilience_kit/recovery.py, RecoveryMonitor.start()
def start(self):
    with self._lock:
        if self._task is not None and not self._task.done():
            return
        # Reassign rather than .clear() so the Event binds to the
        # *current* event loop. asyncio.Event lazily binds to
        # whichever loop first calls .wait()/.set() on it; reusing
        # the prior Event across loops (test harness, restarted
        # server) raises ``RuntimeError: bound to a different event
        # loop``. A fresh Event has no binding yet.
        self._stopping = asyncio.Event()
        self._task = asyncio.create_task(self._run(), name="resilience_kit.recovery_monitor")
    _logger.info("RecoveryMonitor started.")

Six lines changed. The lesson generalizes: asyncio.Event, asyncio.Queue, asyncio.Lock and several siblings capture their loop binding on first use, and .clear() doesn't unbind. Loop-bound state is loop-scoped, not loop-local. Construct fresh on every lifecycle restart; don't cache.

`9f3d846` — `pybreaker` wrapper swallowed the tripping-call exception

The kit wraps pybreaker.CircuitBreaker as one of three breaker backends. On the call that trips the breaker (the one that crosses the failure threshold), pybreaker raises its own CircuitBreakerError to signal "I just opened" — discarding the underlying exception that caused the trip. The caller saw "breaker open" without knowing why it opened. Forensics suffered.

The fix: the wrapper captures the original exception in a small state object before letting pybreaker raise, then re-raises the original on the tripping call. The breaker still opens; the caller still sees the real failure cause. Don't let third-party wrappers eat your forensic signal.

Honourable mentions

aaa5f41 — ASGI header bytes vs str. The request_id middleware did .lower() on b"X-Request-Id". Worked on Starlette's str-typed headers, broke on raw ASGI bytes. Boundary type discipline.
1c8f834 — settings cache leaking the test source between tests, masking what legacy_env_alias actually resolved. The reset helper now restores the default EnvSettingsSource, not the last test's override.
e588e8c — CodeQL flagged a dict whose keys were unioned str | bytes. The fix was three characters. The lesson: take CodeQL findings seriously even when "it works"; the static analyser sees the type-confusion path the runtime tests don't.

These are real fixes from the kit's git history. None of them were dramatic. All of them earned a small piece of the design.

Building the Dangerous Parts First

Two areas in the kit are trivial to get wrong and catastrophic when they fail.

SSRF guard + DNS rebinding TOCTOU

An SSRF guard that just validates a resolved IP is vulnerable to a Time-of-Check-to-Time-of-Use (TOCTOU) race. The attacker's DNS returns a safe public IP at check time; the SSRF guard approves; the underlying HTTP transport does its own DNS lookup; the attacker's DNS returns 127.0.0.1 this time; the request lands on localhost.

The fix is to make the check and the connect share state. AsyncAPIClient resolves the host, validates the IP against the SSRF allow-list, and pins the validated IP into a contextvars.ContextVar. A custom httpx transport reads the pinned IP and connects directly — skipping the second DNS lookup entirely while preserving the original Host header and TLS SNI hostname so cert verification still passes.

The check and the connect now share a single piece of state. TOCTOU bypassed.

Fire-and-forget audit dispatch

Audit logs can't bring down the application. The pre-kit boilerplates wrote audit rows synchronously — if the database hiccupped, the request failed. The kit replaces that with a decoupled, fire-and-forget pipeline:

An in-memory, bounded asyncio.Queue (default 10,000 events) acts as a buffer.
A background worker flushes batches to the configured audit backend (postgres, stdlib_logging, or a custom entry-point backend).
If the database is down, the dispatcher degrades gracefully. It retries with backoff; if it ultimately fails, it logs the batch to stderr and increments a dropped metric. The app survives.

The audit dispatcher is also one of the loop-owned resources from axiom 4 — the Django adapter spawns it inside its daemon thread's private loop and drains it via atexit on the same loop before closing.

Verification: Testing What Unit Tests Can't See

The test suite splits three ways. Unit tests verify logic in isolation. Contract tests verify the same logic works against every backend. Integration tests verify the whole stack survives chaos.

Contract tests: the same assertion, every backend. The kit defines multiple backends — memory, Redis, pybreaker for circuit breakers. The same test suite runs against all of them, parametrized by pytest.mark.parametrize("backend", …). A Lua script in redis_impl.py that diverges from the in-memory contract fails the same assertion that passes against memory_impl.py. The contract suite catches it the moment it ships.

SSRF TOCTOU validation. The integration test (tests/integration/test_dns_rebinding.py) mocks socket.getaddrinfo with a _RebindingResolver that returns a safe public IP on the first call and a private IP on the second. The SSRF guard validates the first IP and pins it. The connection should use only that pinned IP — no second resolution. The test asserts the underlying transport's request URL carries the pinned IP, not the attacker's late-arriving private one.

Recovery monitor validation. The integration test (tests/integration/test_recovery_monitor.py) spins up a real Redis with testcontainers, calls docker_client.api.pause(container_id) to freeze TCP without killing the connection, verifies that throttles degrade to in-memory and flag degradation in metrics, then unpauses and verifies the monitor restores the Redis-backed providers within the 5-second exit-gate window. This test is what caught the bc499da bug — the second invocation of the monitor across pytest's per-function loops blew up exactly as described above.

The kit ships with 52 test modules across unit, contract, and integration suites.

What I'd Do Differently

Four regrets from the critical path:

Chaos and failure injection should come before v0.2. I built the core under happy-path assumptions, then spent the adapter phase hardening it. Testcontainers, container-pause cycles, Redis-down scenarios — these should have been mandatory from day one, not post-release validation. Every failure mode I caught during adapter work would have reshaped a primitive's design if I'd seen it earlier.
CI matrix (Python 3.11, 3.12, 3.13) should be commit-one, not late hardening. I assumed 3.11 compatibility, then discovered version-specific differences in asyncio behaviour and stdlib deprecations during release prep — exactly the work a GitHub matrix would have done on every PR for free. Fixes were mostly mechanical; the cost was the discovery cycle.
Contract test suite should precede any backend implementation. I wrote backends (memory, Redis, pybreaker) and then wrote tests to validate them. Tests written after backends encode the backends' assumptions as the spec, not the spec's expectations as the test. Contract tests written first force you to commit to the behaviour before you commit to the implementation — and they catch backend-specific divergence at the boundary rather than during integration. I got there eventually; getting there first would have been cheaper.
mypy --strict should be enforced from commit-one, not applied retroactively. I shipped the first half of the project without strict type hints, then made it a release gate. Retrofitting types to async code with complex control flow is error-prone, and the rewrite surfaced structural mismatches between Protocol declarations and the backends meant to satisfy them. Strict types from the start force the design right the first time.

These are design regrets, not dogfooding bugs. The dogfooding bugs fed v0.1.1 and v0.2. The design regrets are about how I'd budget time on the next from-scratch infrastructure project.

What's Next

The convergent v0.2 wishlist from both migration reports is half-written for me:

DjangoSettingsSource — read settings.RESILIENCE directly instead of forcing env-only config. Both reports rank this P1.
resilience_kit.utils.* — five small modules (log_sanitization, network, timing, function_logger, data) every boilerplate re-implements. Promoting them into the kit saves ~900 LOC per consumer.
MetricsSink cardinality contract — bound the labels at record_* time, not after a Prometheus cardinality explosion in prod.
GlobalThrottle (Valkey-Lua) — process-wide cap. Nginx covers it for fleet deployments; the in-process belt is for laptop dev and single-pod deploys.
Multi-alias Redis topology — RESILIENCE_REDIS_URLS__<alias> so cache, throttle, and breaker can point at separate instances.

The v0.2 exit gate is the same rule, raised: both boilerplates re-test against a v0.2 pre-release and score ≥ 8.5 / 10 — half a point higher than 0.1.0.

Full roadmap: ROADMAP.md.

I started by trying to delete duplication. I ended up with five axioms, a dogfooding release rule, and a set of bug-earned invariants that are reusable far beyond this kit. The duplication was the symptom. The axioms are the fix.

Notes & references

Each axiom and dangerous-parts decision has a one-page ADR in the repo. If you want the long-form reasoning — alternatives considered, consequences, usage notes — these are the load-bearing ones:

Topic in this post	ADR
Protocols, not ABCs	`0001-protocol-not-abc`
Hand-rolled retry over `tenacity`	`0002-handrolled-retry-not-tenacity`
One package, many extras	`0003-single-package-with-extras`
Entry-point backends + precedence chain	`0004-entry-points-for-third-party-backends`, `0009-entry-point-precedence-chain`
Fire-and-forget audit dispatch	`0005-fire-and-forget-audit`
Outer breaker, inner retry	`0006-outer-breaker-inner-retry`
DNS pin via `ContextVar` (TOCTOU)	`0007-dns-pin-via-contextvar`
Fernet env-guarded keys	`0008-fernet-env-guard`
FastAPI adapter shape	`0010-fastapi-adapter-shape`
Django sync/async bridge	`0011-django-sync-async-bridge`

Migration reports: docs/m8b-upgrade-reports/ — SUMMARY.md, fastapi_boilerplate.md, django_boilerplate.md.

The adapter is not the kit. The kit is what survives without the adapter.

Source Code: resilience-kit GitHub Repository
Documentation: Design Doc (PRD) | Architecture (LLD) | ADR Index | Migration Reports

Building toymq: a from-scratch persistent message broker in Go

Prajwal Mahajan — Tue, 09 Jun 2026 20:04:45 +0000

A retrospective on toymq — a single-node persistent message broker in Go, recreated by hand to understand one of the smallest functional units in a distributed system.

TL;DR — and the bug you should remember

Build the dangerous parts first. Build the thing that, if it's wrong, forces a rewrite of everything above it. Then build the thing above it.

The worst bug of the project took six commits to surface and an integration test to catch: a uint64 consumer offset where 0 meant both "message id 0 was acked" and "never acked." Unit tests never restarted the broker between Ack and Subscribe, so the collision stayed invisible. Zero values are not sentinels. That's the lesson the post earns.

The numbers: ~10k lines of Go, 17 ADRs, four binaries, 90.3% test coverage, a chaos harness that pushed 14,000 messages through three SIGKILL/restart cycles with zero acked-message loss. v1.0 in four days; v1.3 plus post-release hardening in another two.

The order:

WAL → wire protocol → broker → session → server → cmd wiring
   → integration tests → chaos → post-v1.0 hardening

The point of this post is the order, not the code.

What toymq is, and isn't

A learning artifact. Single node. No replication. No auth. No TLS. Throughput tops out at a few hundred messages per second because every publish does a per-message fsync. If you need a real broker, reach for NATS or RabbitMQ. If you want to understand what a broker is — what durable actually means, what at-least-once costs, what crash recovery requires — build one. That's what this post is about.

Why "build the bottom first" is non-obvious

Every tutorial does the opposite. You build a server, accept connections, parse a request, stub a handler, add real handlers, add persistence last. By the time you reach durability, you've shipped a wire protocol that assumes you don't have one, a handler API that assumes nothing crashes, and a test suite that mocks the part of the system that matters.

toymq inverts that. The first commit isn't a server — it's a framed record format and a CRC check. Day 1 is "what is on disk after a crash" and nothing else.

This works for a selfish reason: the bottom of the stack, if I get it wrong, forces a rewrite of everything above it. A storage bug means the broker is wrong. A protocol bug means every client and the broker are wrong. Risk shrinks as you go up. So you spend the budget for being wrong at the bottom, while the budget is highest.

Durability: what should "on disk" actually mean?

The storage piece of any persistent system is the write-ahead log — a file you append records to, fsync, and never go back to mutate. On crash, you scan it from the start and rebuild your state.

The one rule for the WAL: the on-disk format is a contract. Anything that lands on disk has to survive forever, because the recovery scan will assume it. Changing the format later means writing a migration.

On-disk layout

data/
└── topics/
    ├── orders/
    │   ├── segment.log     ← append-only WAL, one record per PUB
    │   └── offsets.json    ← per-consumer state, written by debouncer
    └── events/
        ├── segment.log
        └── offsets.json

One directory per topic. Two files per topic. No metadata file, no index, no manifest. The recovery scan walks every segment from offset zero on Open. Scans are O(disk) but correct by construction — the WAL is its own truth, and a manifest would be a second consistency problem on top of the first.

The record frame

Field	Type	Size	Notes
`length`	`u32`	4 bytes	Bytes that follow, excluding self
`msg_id`	`u64`	8 bytes	Monotonic per topic
`ts_ns`	`u64`	8 bytes	Append timestamp, ns since epoch
`key_len`	`u16`	2 bytes	0 if the message has no key
`key`	bytes	`key_len` bytes	Dedupe key
`payload_len`	`u32`	4 bytes	Payload length in bytes
`payload`	bytes	`payload_len`	Message body
`CRC32`	`u32`	4 bytes	Checksum over all preceding bytes

Deliberately boring. The interesting part is what's not there: no version byte. Adding a version byte later is a one-line migration; defending the wrong version byte forever is not. Don't add knobs the format doesn't yet need.

`fsync` is the durability commit point

Log.Append is the only function in toymq that calls fsync. Every PUB goes through it. Every OK the broker emits is, by construction, preceded by a successful fsync of the corresponding record:

The committed-offset update happens after fsync. Any reader that sees committedOffset == X is guaranteed the bytes through X are on stable storage. Cost: ~1–2 ms p99 on commodity NVMe. The correctness budget gets spent here, not on throughput tricks I can't defend. A broker that loses an acked message is not a broker.

How this compares to Kafka

Kafka does almost none of this. Its durability story is page cache + replication: trust the OS to flush eventually, trust the replicas to catch up. It pushes hundreds of thousands of messages per second per broker as a result. toymq has neither replicas nor the throughput budget to assume the page cache wins, so it trusts fsync directly. Different problem, different answer — and the gap between those answers is exactly what hand-rolling teaches you.

The first nasty bug landed here, too: a make([]byte, payloadLen) that ran before checking payloadLen <= maxPayload. A malicious client could OOM the broker with a single packet claiming a 100 GB payload. The fix is one line; the rule is general — validate before allocating, always.

The wire protocol: agreeing on what happened

Once the storage layer holds, the next question is how clients and broker talk. A protocol is the contract that lets the two ends disagree about when something happened (network is unreliable) without disagreeing about whether it happened.

toymq's protocol fits in a paragraph: a command word, a few arguments, a length-prefixed payload. The full PUB happy path, end to end:

The OK only crosses the network after fsync returns. That's the durability promise made visible: the client never sees an OK for a message that isn't on disk.

Two design choices shape everything above this layer:

A sealed Command type. The parser dispatches on an interface with an unexported marker method. You cannot add a new command type without editing the file the parser lives in. The compiler keeps the protocol and its handler in sync — if you add a command and forget to handle it, the build breaks. Free safety.

Clean EOF vs torn header. The parser distinguishes a stream closed cleanly between commands (propagate io.EOF) from a stream closed mid-line (return ErrBadFraming). That distinction recurs everywhere downstream: the session loop uses it to tell "client gracefully disconnected" from "client crashed and dropped." Most tutorials skip this and pay for it forever.

Brokering, in two parts: structure vs semantics

A broker has to do two things: route messages from producers to consumers, and guarantee something about the delivery (at-least-once, in toymq's case). I split this into two branches and the split mattered.

Structure came first: topics (auto-created on first publish), a WAL per topic, an LRU dedupe index keyed by (producer-id, msg-id), in-memory consumer state. Acceptance bar: "you can publish a message and consume it; if the broker restarts, recovery works." No visibility timeouts, no NACKs.

Semantics came second. Every message a consumer has been told about lives in one of three states:

A message can bounce between Pending and Inflight many times before reaching Acked — that's the at-least-once contract. The chaos suite's no-loss invariant verifies that every acked MsgID reaches at least one consumer's seen set. Duplicates above one are allowed; loss is not.

The redelivery ticker is the load-bearing piece, and it forced the rule that became the hot-path discipline for the entire broker:

Never send while holding the inflight lock.

c.inflightMu.Lock()
pending := append([]*Message(nil), c.inflight...)
c.inflightMu.Unlock()

for _, msg := range pending {
    select {
    case c.SendCh <- msg:        // safe — no lock held
    case <-ctx.Done():
        return
    }
}

A blocking send on a channel, while holding a lock the redelivery scan also wants, is deadlock bait. Take the snapshot under the lock, release it, then send. The symmetric rule on the ticker side: the scan does hold the lock for the full pass — releasing per-entry lets a concurrent ACK delete an entry mid-iteration and panic the map. -race finds the second one in five seconds; code review catches the first.

I tried to do structure + semantics in one branch first. I rolled it back at the second self-merge conflict. The two-merge rule: if a branch conflicts against itself twice before it's done, the branch is too big. Split it.

Concurrency: serving many connections without corruption

The broker is one process; clients are many. The design question is how many goroutines per connection, and who owns what.

toymq's answer: three goroutines per session, one channel of truth between them. A fourth — the broker's per-subscription delivery worker — lives on the broker side but writes into this session's outbound channel.

Session R owns the socket's bufio.Reader. Nothing else reads.
Session W owns the socket's bufio.Writer. Nothing else writes.
Session H dispatches parsed commands to the broker and feeds responses into outbound.
runDelivery is the broker's delivery goroutine for this consumer; it also writes into the session's outbound channel.

outbound is the only place where two goroutines (H and runDelivery) might race to write the socket — and the channel itself serializes them, no lock needed. bufio.Writer has no concurrent-safe contract, and you don't lock around it; you funnel through a channel. Closing the socket, draining outbound, and joining all goroutines becomes one operation: cancel a context, wait on a WaitGroup.

The rule that came out of this is sharp and reusable: every go-spawned goroutine must be accounted for in exactly one WaitGroup or doneCh. A "WaitGroup race" in the listener — wg.Wait returning before a goroutine that called wg.Add had a chance to register — was the bug that taught me to write the rule down explicitly.

The TCP accept loop adds two subtleties most servers get wrong: an EMFILE-aware exponential backoff (a burst of connections can exhaust file descriptors and tight-loop the broker), and an explicit listener.Close() on context cancel (without it, Accept() blocks forever and shutdown hangs).

cmd/toymq keeps main.go thin: a separate internal/config package owns flag validation, a testable run(ctx, args, stdout, stderr) int lets tests drive shutdown via context cancellation, and signal.NotifyContext wires SIGTERM/SIGINT into the context in one line. The smoke test boots run, sends a fake SIGTERM, and asserts a clean exit. It catches "shutdown leaks a goroutine" the moment it happens.

Testing what unit tests can't see

By this point every package had unit tests. Coverage was 78%. -race clean. But the most interesting bugs in any networked system live in the seams — what happens when the broker sends a 300-byte MSG, the client receives bytes 1..200, and then disconnects?

I built in-process integration tests next: a real broker + server on a kernel-assigned TCP port, a stripped-down test client that queues incoming MSGs so asynchronous deliveries don't deadlock the response-reading. Six scenarios: round-trip ACK, 1000-message restart, visibility-timeout redelivery, NACK redelivery, dedupe, subscribe takeover. Coverage went from 78% to 90.3%.

The suite caught the worst bug of the project on its first run. The test:

Publish message id=0. Subscribe. Receive MSG. ACK.
Restart the broker.
Subscribe again. Expect no replay.

It failed. Message id=0 replayed. With id=1 as the first message, no replay. The bug was specifically the zero msg-id.

Root cause: consumer state on disk stored a single lastAcked uint64. Recovery treated lastAcked == 0 as "never acked" (the Go zero value) instead of "acked msg id 0." On the first restart after acking the very first message, the broker couldn't tell the difference. The fix adds a hasAcked bool and persists both:

type ConsumerState struct {
    LastAcked uint64
    HasAcked  bool   // ← the field that should have been there from day 1
}

The Go-language lesson: zero values are not sentinels. They collide with valid data the moment your domain is non-empty. The systems lesson is sharper — the bug existed for six commits before anyone noticed. Unit tests never restarted the broker between Ack and Subscribe. By the time integration tests ran, the broken state was sealed in code-review history weeks earlier. The integration test caught it on its first run.

Then I added chaos: a supervisor that SIGKILLs the broker on schedule and restarts it; a producer that keeps publishing through restarts; a consumer that records every msg-id it ever sees. Invariant: for every msg-id the producer got an OK for, the consumer must receive it at least once, eventually. A 90-second smoke pushes 14,000 messages through three SIGKILL cycles with zero acked loss. That's not a proof of correctness — it's evidence the WAL + fsync + offset design holds up under realistic crash patterns.

Chaos also found a data race in the chaos test itself — a bytes.Buffer capturing stderr being written by the supervisor and read by the assertion. The broker was correct; the test had the race. -race is for everything you ship, including tests. A race in test infrastructure can mask real bugs or manufacture phantom ones.

After v1.0: the part where the work gets easier

Once the bottom half held, the top half was craft, not danger. A Go client library (pkg/client) on top of the protocol, with a single read goroutine that demuxes incoming frames by type — separate channels per call would have raced two reads on the socket, a deadlock recipe. A CLI (toymqctl) and a latency harness (toymq-bench). A Bubble Tea TUI (toymq-tui) with one sharp lesson worth keeping: boolean state for "is X active?" is a smell. A bool for "is a modal open?" broke the day the second modal shipped on top of the first. A modal stack fixed it. Lists, stacks, enums — anything but a boolean — almost always model the actual shape.

Then Prometheus metrics and OpenTelemetry tracing, with the design rule: observability does not change behavior. No locks, no goroutines, no allocations on the hot path. prometheus.CounterVec.WithLabelValues is cached; otel.Tracer().Start is a no-op when no provider is attached.

Finally, CI hardening after main went red twice in a row with gofmt drift. Two recurrences is the line where you stop fixing instances and start fixing the system: a Makefile, an opt-in pre-commit hook, golangci-lint with a conservative ruleset, and a Go 1.25 + 1.26 CI matrix. The linter's first run flagged 31 things; seven were real bugs (commited, exhuastion, three dead helpers, one cap-shadowing parameter). A linter that flags 30 things on its first run is doing its job.

ADRs as crystallization, not ceremony

Seventeen ADRs sounds like ceremony. It wasn't.

The rule: write the ADR the moment the decision is forced by code, never before. Not when I had a hunch. When I had to type "we are doing it this way" into a function, then the ADR went next to it.

ADRs written before code force the code to fit the ADR, which means defending decisions made when you knew the least. ADRs written at crystallization record what the code already decided. If the code later changes its mind, you supersede the old ADR — you don't try to retroactively justify it. The point is that the ADR is always honest about when the decision was made.

What I'd do differently

Run chaos one branch earlier. Integration tests didn't find what chaos found one branch later. Chaos was the right tool, just deployed one branch too late.
CI matrix from day one. A Go 1.25 + 1.26 matrix takes 30 minutes to set up and prevents an entire class of "works on my machine" failure. It should have been part of the first chore(ci) commit, not a retrofit at the end.
Wire trace context through the protocol from v1.3. Spans today are root spans inside the broker. Cross-process correlation needs a TRACEPARENT line in the wire format. Wire-format changes are exactly what you regret deferring.
Treat zero values as a code smell from day one. The lastAcked bug was the most embarrassing of the project. Whenever the answer is "use 0 / "" / nil / -1 to mean absent," the right answer is (value, present bool).

What's next

Persisting the dedupe LRU to disk without losing the boring-by-design property of the WAL. Right now the LRU is in memory only — survives one broker lifetime, lost on SIGKILL. The fix mirrors the atomic-swap pattern used for offsets; the interesting question is what happens on a torn write of the LRU snapshot itself, since unlike the WAL there's no CRC-framed record stream to scan. That's the next post.

After that, the series moves to tinykv (a Redis-subset KV store in Go) and tinyraft (a 3-node Raft consensus cluster). Same risk-first sequencing. Same ADRs-as-crystallization. The arc ends where you compose them: replicated state machine semantics on top of a real consensus log.

The one-line version

If a tutorial tells you to build the server first, you've already started in the wrong place. The storage is the project.

Source: prajwalmahajan101/toymq. Deeper docs (with every diagram in this post and more): ARCHITECTURE.md, PERSISTENCE.md, CONCURRENCY.md, REDELIVERY.md, FLOWS.md. ADR index: docs/adr/. Open ideas: IDEA.md. Corrections welcome — open a discussion or file an issue.

DEV Community: Prajwal Mahajan

Building toykv: a from-scratch persistent KV in Go, and why I took the opposite call from toymq three times

TL;DR — and the rule I want you to remember

What toykv is, and isn't

The crash matrix

Inversion 1 — a version byte you'll thank yourself for, and absolute deadlines

Absolute deadlines, never relative durations

The AOF on disk

Inversion 2 — write the bytes you'd send

Inversion 3 — RESP because the harness is free

Honourable mention 1 — the dual-write bufio bug

Honourable mention 2 — the test was wrong, not the code

ADRs as crystallization — and the ADR I almost didn't write

What I'd do differently

What's next

The one-line version

Building resilience-kit: A Python Resilience Kernel Forged in Production

TL;DR

Five Axioms

1. One package, many extras. Not a forest of micro-packages.

2. Protocols, not ABCs.

3. Outer breaker, inner retry. Composition order is correctness.

4. Async-first with explicit sync bridges. Never trust auto-bridging.

5. Adapters are dumb wiring. If your adapter has business logic, your primitive is wrong.

The Dogfooding Gate

What the Migration Reports Found

Bugs the Kit Found in Itself

f6cebdd — outer breaker, inner retry

bc499da — asyncio.Event rebinding on every restart

9f3d846 — pybreaker wrapper swallowed the tripping-call exception

Honourable mentions

Building the Dangerous Parts First

SSRF guard + DNS rebinding TOCTOU

Fire-and-forget audit dispatch

Verification: Testing What Unit Tests Can't See

What I'd Do Differently

What's Next

Notes & references

Building toymq: a from-scratch persistent message broker in Go

TL;DR — and the bug you should remember

What toymq is, and isn't

Why "build the bottom first" is non-obvious

Durability: what should "on disk" actually mean?

On-disk layout

The record frame

fsync is the durability commit point

How this compares to Kafka

The wire protocol: agreeing on what happened

Brokering, in two parts: structure vs semantics

Concurrency: serving many connections without corruption

Testing what unit tests can't see

After v1.0: the part where the work gets easier

ADRs as crystallization, not ceremony

What I'd do differently

What's next

The one-line version

Honourable mention 1 — the dual-write `bufio` bug

`f6cebdd` — outer breaker, inner retry

`bc499da` — `asyncio.Event` rebinding on every restart

`9f3d846` — `pybreaker` wrapper swallowed the tripping-call exception

`fsync` is the durability commit point