DEV Community: Prajwal Mahajan

Building toykv v2: seven features that didn't earn the major, one refusal that did

Prajwal Mahajan — Wed, 22 Jul 2026 10:34:13 +0000

A retrospective on the v1 → v2 arc of toykv — the jump from a single-node, RESP2-compatible KV that works on localhost to one that's safe to bind off it. The first post was about earning a durability contract and taking the opposite call from toymq three times. This one is about adding seven features to that store without betraying the two contracts v1 earned — byte-compatibility on the wire, and durability under crash — and then discovering, in the phase I'd budgeted for polish, that the scariest bugs of the whole arc were still hiding.

TL;DR — the decision, and the check, you should remember

Part 1 built the dangerous part first: AOF → replay → RESP wire, with one rule — absolute deadlines, never relative durations — and a nine-row crash matrix proving it held. v1.0.0 shipped a KV that survives SIGKILL with zero acked-write loss, speaks enough RESP2 that redis-cli and go-redis are free test harnesses, and does all of it on localhost with no auth.

v2 is the deployment problem. The store is correct; the job is to make it reachable — RESP3, lists and hashes, AUTH and TLS, INFO and SCAN, a rebuilt TUI, atomic keyspace ops, OpenTelemetry — without regressing byte-compat or durability, and without pretending the version number is honest when it isn't.

Here's the uncomfortable thing about that version number. By pure semver, the whole feature set is a 1.x. RESP3 is opt-in behind HELLO 3; RESP2 stays byte-identical. The new AOF format (v3) replays v1 and v2 files unchanged. AUTH and TLS are off by default. Observability with no endpoint is a no-op. Every one of those is additive — a pre-v2 client is served exactly as before. Adding features, however many, does not earn a major.

The one thing that makes 2.0.0 honest is protected mode: v1 would bind 0.0.0.0 with no auth and serve the internet a plaintext database. v2 refuses to start that configuration — the only line in the whole arc a pre-2.0 operator can't ignore. That's the decision to remember: you don't earn a major by adding features — you earn it by changing a contract, and the only contract change in v2 is a server that says no. It's the toykv-shaped sibling of what toymq v2 calls "the durability contract is sacred" — same series move, different axis: toymq protects durability; toykv protects byte-compat + safe-by-default.

But the check to remember is the one that nearly cost me. The final phase was scoped as "bench + polish + tag." It's where I found the two scariest bugs of the arc — a pre-auth DoS a single packet could trigger, and a 20% regression in the "free" telemetry path — and then, after tagging, two more: a RESP3 codec that could encode but not decode its own frames, and hash commands returning fields and values from two different random iterations. None were caught by a suite green across a six-way OS/Go matrix. All were caught by running something for the first time.

That collapses to a one-liner that belongs to a family. Each post in this series earns one the same shape:

Post	The hero rule	The trap it names
`toymq` v1	zero values are not sentinels	`lastAcked == 0` meant both "acked id 0" and "never acked"
`toykv` v1	durations are not deadlines	`EX 5` on disk re-applies against `now` after every restart
`toymq` v2	derived state is not a source of truth	a `dedupe.json` sidecar re-materialises the duplicate it was meant to kill
`toykv` v2 (this post)	a green check is not a verified one	a suite green on the wrong reader, a `t.Skip`ped compat sweep, and a benchmark nobody ran all show the same green

Every rule in that table is the same shape: a thing that looks authoritative until a restart, a bind, or a first real run calls its bluff. A zero looks like a sentinel; a duration looks like a deadline; derived state looks like truth. And a green checkmark looks like verification — right up until you notice the test skipped, ran against the wrong surface, or was never run at all.

The rule: a roadmap — and a CI dashboard — records intended state, not verified state. Green is not verified. The only check that verifies anything is the one you actually ran, this session, against the surface a user actually walks.

The numbers: ~5k LOC grew past 12k, seven design decisions recorded (one amended after release — the supersede trail below), the AOF at format v3, four binaries still, and a 2.0.0 whose CHANGELOG cut is dated 2026-07-18. The order:

RESP3 → lists/hashes + AOF v3 → AUTH/TLS → INFO/SCAN
   → TUI rebuild → protected mode + atomic ops → OpenTelemetry → release

Six additive steps in blue, amber, and purple; one red break in the middle. v1's rule was "risk sinks down — build the AOF first." This is that rule rotated: with the store frozen, risk drains outward from the wire, and the single red node is the only thing a pre-2.0 user feels. The point of this post, like the last one, is the order and the rules — not the code.

Why the order is the design

With risk draining outward from the wire (above), three ordering principles did the work.

The wire foundation lands before the thing that needs it. RESP3 ships before the typed values, even though it has no typed values to carry yet, because HGETALL → map is the first real consumer of its frames — the same bottom-up logic that put the AOF before TTL in v1.

The deployment ceiling lifts before the thing that needs it lifted. Auth and TLS make the server safe off-host; SCAN and the TUI are what you'd actually use off-host. A browsable KV you can't safely bind is a tool for nobody.

The one break clusters behind the major bump. v2 has exactly one break a pre-2.0 user feels — protected mode refusing an unsafe bind. Everything else is additive by construction, so the major number rides on a single, nameable, defensible change.

The rule: you get one contract break per major version. Spend it on the one thing that makes the version honest — for toykv, a refuse-to-start — and keep everything else additive so a v1 user upgrades without noticing.

And each feature owns its risk test, written where the risk was introduced — no "integration pass at the end" that discovers the value-types crash bug during the observability work. Which is exactly why it stung when the release pass found bugs anyway: it was the first time anyone tested the seams between features, and the seams were where the bugs lived.

Seven rules v2 earned

Like part 1, I didn't write these up front. I read the decision records back and asked what did I refuse to compromise on?

A major is earned by changing a contract, not counting features. Five additive features don't add up to a 2.0.0 — they add up to a fat 1.x. Protected mode is the single behavioural break, and its decision record says so: "one deliberate breaking change plus one correctness win, not feature-count padding."
Additive means provably additive — the downgrade point is where you prove it. Handlers return protocol-agnostic resp.Values and a single writer function downgrades each rich kind to RESP2 — map→flat array, null→$-1. The compat sweep asserts every command's exact RESP2 bytes, so "unchanged" is enforced, not hoped. HGETALL returned resp.Map(...) and changed zero writer code.

The version byte's hardest case is the old file you're about to append to. Everyone bumps the format for new files; the sleeper is the existing v2 file the new binary appends to. The instant an LPUSH lands in a file whose header says 0x02, the header lies. So Open() pwrites the version byte at offset 7 and fsyncs before any append — a file's header version ≥ its newest record format at every instant.
The durability contract is still sacred — and a feature is the most likely thing to break it. The aof.append span is a pure outer wrapper: a slow fsync shows as latency, but the span can't reorder a byte inside the writer. Same lesson toymq v2 learned from the other side — once you've earned durability, the thing most likely to break it isn't hardware; it's the next feature.
Constant-time where it's secret, plain compare where it isn't. AUTH uses crypto/subtle.ConstantTimeCompare on the password, a plain compare on the public default username, and returns the identical -WRONGPASS invalid username-password pair or user is disabled. for a wrong user or a wrong password — no oracle for which half failed.
Off-by-default has to be free, and "free" is a benchmark, not an assumption. The observability layer is a no-op by construction, and I asserted that made it free. It didn't (see the release pass, below): building an attribute.Set allocates whether the SDK records it or not — off by ~20% until measured.
In-memory derived state keeps the on-disk format frozen. SCAN's cursor is a never-persisted per-key seq, re-derived in record order at replay; RENAME/COPY record verbatim under the v3 reader. Neither bumped the AOF. A format bump is a liability you pay forever; a derivation is a loop you pay once at Open.

The dangerous parts of v2, one at a time

Lists and hashes — the header byte you rewrite under the old binary's feet

Adding lists and hashes meant the store's value became a tagged union (typ + three concrete fields, no any, no interface — so no type assertion can panic), and the AOF took its second format bump to v3. The union was mechanical. The dangerous part was the header upgrade described in rule 3.

The O_APPEND gotcha en route is the kind of thing you only hit by doing it: the live append fd is opened O_APPEND, and Go rejects WriteAt on an append-only handle. So the one-byte header rewrite needs its own short-lived, non-append handle, used and closed before the append path takes over. The restart probe after a fresh dir:

header bytes: 54 4f 59 4b 56 00 00 03   →  "TOYKV\0\0" + version 0x03

The crash test for this work is model-based, not a spot check: the parent keeps its own model of acked mutations, SIGKILLs the child mid-stream of mixed list/hash ops under fsync=always, restarts, and asserts the replayed store equals the model exactly, list order included — the v1 durability discipline extended to the surface that could newly corrupt it.

Honourable mention — the "flaky" test was telling the truth

This one surfaced during the auth + TLS work, because that's where CI caught it, but it's a v1 bug that had been flakily caught since the live-rewrite path landed. CI went red: TestAOF_CrashInjection_DuringRewrite/mid-kill, "2/82 acked SETs lost across rewrite+SIGKILL." The roadmap's release gate had this exact test pre-authorized for quarantine — "flaky, fix or quarantine." It would have been so easy to quarantine it.

The failure signature was the entire diagnosis. The lost keys were always live000 and live001 — the first two live writes during the rewrite, never random ones. A timing flake loses arbitrary keys; losing the earliest is an ordering bug. The mechanism is the diagram below: the rename unlinks the old inode before the side buffer is drained, so in the [rename, drain] window an acked write lives only in the doomed inode plus a volatile buffer. -race on CI widened the window enough to hit it; locally it passed 15/15 without -race.

Two things made it into the lesson. The decision record had rationalized the bug — the original rewrite design documented that window and called it "no worse than the single-file baseline," which was wrong: v1's single-file path never unlinked live data mid-write, and a crash-invariant table with a data-loss row marked acceptable is a design smell in a durability-first project. A decision record written after the code can launder a bug into a decision. Meanwhile the low-level design had it right all along — it specified fold-then-rename; the code shipped the reverse.

The fix (901806d) folds the side buffer into .tmp and fsyncs it before the rename, holding the append lock across the whole capture → rename → fd-swap. And because I couldn't reproduce the SIGKILL flake locally, I proved the mechanism deterministically: an in-process regression test with an afterRenameHook that replays the canonical file at the exact rename instant — it recovers only the snapshot and fails loudly on the old ordering, passes on the fix, 0/30 under -race. A probabilistic crash test became a deterministic ordering assertion.

The rule: read the failure, not the "FAIL." A crash test that loses always the earliest writes isn't flaky infra — it's an ordering bug wearing a probability distribution. And when you can't reproduce the race, model the instant the crash would expose and assert the invariant there.

AUTH + TLS — the feature that shipped in one boolean

The RESP3 work paid a churn I chose deliberately: every handler gained a *connState parameter most of them ignore (_ *connState). It looked like noise. It was the seam. When auth arrived, connState gained one bool, dispatch gained one condition, HELLO's stub branch became a real check, and the listener got wrapped in tls.NewListener. Zero handlers touched. The plan's "verified against live code" pass even found conn_state.go's doc comment already reading "authentication state lands here" — pre-plumbed a release early.

Two ordering calls carried weight. Gating goes before the command-table lookup, so an unauthenticated NOSUCHCMD gets -NOAUTH, not -ERR unknown command — the table leaks nothing pre-auth. And the no-auth path is the degenerate case: newConnState(id, s.cfg.RequirePass == "") authenticates every connection of an auth-less server at accept, so v1 behaviour is provably identical — the same trick that auth-exempts AOF replay by construction.

SCAN — you can't port Redis's cursor to a Go map

SCAN reads like a command handler. The actual work was a store-model decision the language forced. Redis's cursor is reverse-binary iteration over its open-addressed hash buckets — it survives rehashing and gives the "every key present for the whole scan is returned" guarantee. A Go map[string]entry randomizes iteration and exposes no buckets. Redis's technique is simply unavailable.

The answer: stamp each entry with a monotonic seq at creation, preserve it across every update, and make the cursor a seq value. Scan returns live keys with seq > cursor sorted by seq; a key alive for the whole scan keeps its seq and is always reached. The subtlety is that seq survives updates but not re-creation — and the code nearly made that free, because read-modify-write paths copy the whole entry back, so seq rides along; only four sites build a fresh entry, and one grep for entry{ up front meant the invariant held on the first test run. The owned risk test seeds low-seq noise, then the higher-seq must-see set, then deletes the noise while scanning — the exact index-shift a positional cursor fails and the seq cursor survives.

INFO taught the adjacent lesson. The roadmap said "RESP3 map when the client is on RESP3" — but real Redis returns INFO as a bulk/verbatim string, never a map, and both go-redis .Info() and redis-cli info parse the # Section\nkey:value text. A map breaks both. I surfaced it as a decision and corrected the roadmap. A spec can be wrong; comply with reality, not the doc.

Protected mode — the refusal that earns the major

Everything above is additive. This is the break. Protected mode refuses to start — not to serve — when bound to a non-loopback address with neither requirepass nor TLS. The clean call was putting checkProtectedMode inside server.New rather than inline in main: unit-testable in isolation, it protects any embedder, and it runs before AOF replay so an unsafe bind never touches disk.

The API-shape detail that makes it safe by default is invisible until you construct the struct directly: Config.ProtectedMode is a string whose zero value "" means enabled. A bool would default to false = disabled, so a bare Config{} would be unsafe — the opposite of the point (the decision record lists exactly that as the rejected alternative). It rhymes with the sibling project's "zero values are not sentinels": here the zero value has to mean the safe thing, so I chose a type whose zero value does.

The other half of this work — atomic RENAME/RENAMENX/COPY, a correctness win that rides v3 verbatim (rule 7) — produced the small relatable gotcha. The plan said "reject Redis's DB option outright." Then the go-redis round-trip failed: go-redis v9's Copy always sends COPY src dst DB 0. The Redis-faithful fix — accept index 0 (toykv's single DB), reject others with -ERR DB index is out of range — is both more correct and unblocks the client. Verify the client's wire bytes, not just your server's spec, before deciding what to reject.

Observability, then release — no-op providers are not free, and the auth review missed a pre-auth DoS

The observability layer shipped with the no-op-by-construction design from rule 6, plus a churn call worth naming: the instrument inventory implied threading context.Context into ~30 store methods to parent the store.<op> spans — ~250 edits, almost all in tests, for byte-identical telemetry. So the span is created at the server→store boundary instead: same trace tree, ~0 test churn, store package stays context-free. The means (ctx in store) was never the goal (the span).

Then the release pass — "bench + polish + tag" — turned into the most instructive phase of the arc, because it was the first time anything was verified rather than built. Two gate items marked green were regressions:

A security review scoped at auth found a pre-auth DoS one layer below it. The review of the auth/TLS/protected-mode surface came back clean — but it looked under the auth gate, at the RESP codec that runs before it, and found readArray doing make([]Value, n) with n bounded only against negatives (a *1000000000\r\n header is a multi-GB OOM from one packet) and unbounded recursion depth (a nested-array stack-exhaustion fatal panic you can't recover). Both reachable before any credential, defeating the "safe-by-default" claim that earns 2.0.0. Fixed in ef8a423: MaxArrayLen = 1<<20 (matching Redis's proto-max-multibulk-len), MaxDepth = 32, both rejecting with ErrTooLarge before allocation.

The rule: a clean bill on the named surface is not a clean bill on the reachable surface. The auth review found the DoS precisely because it didn't stop at auth — it asked "what runs before this?"

The "free" telemetry path cost 20%. The parity benchmark existed (BenchmarkObserveCommand_Disabled, 29 allocs/op in isolation) — but nobody had run the actual A/B against the pre-observability binary (a3d62e1). Doing it: SET −21%, GET −18%, every round. Root cause: building an attribute.Set allocates against no-op providers too, so observeCommand rebuilding options per command taxed the disabled path. The fix memoized per-command attributes in a cmdInstr cache — no if enabled guard added — dropping the path to 14 allocs/op, 1757 ns/op, SET back to parity.

Both would have shipped in 2.0.0 if I'd trusted the roadmap's "gates green." That's the headline of the whole arc: the phase I expected to rubber-stamp is where I found the scariest bugs, because it was the first time I checked instead of built.

After the tag — writing the manual to test it found two more

The sequel to that lesson is that even the release pass didn't finish the verification. I wrote docs/TEST_CASES.md as a "local testing guide," and running it by hand against a live server surfaced two more already-shipped defects the six-way CI matrix never caught.

The RESP3 codec was asymmetric. The writer encoded all seven RESP3 kinds (% ~ , # _ = >); ReadFrame decoded only RESP2. internal/client uses that reader — so HELLO 3 under toykv-cli/toykv-tui died on the first RESP3 reply with resp: unknown prefix '%'. The server's RESP3 was correct and go-redis round-tripped fine in e2e, because every existing test exercised a reader that wasn't ours. The missing gate was obvious in hindsight — an encode→decode round-trip over the codec's own two halves — and because it never existed, the asymmetry was invisible. And "decoded" wasn't "done": after the reader fix, the CLI printed (unknown kind '%') because internal/respfmt couldn't render the map. A ninth change I'd never have found without running the binary.

HKEYS/HVALS/HGETALL didn't correspond. The hash was a bare map[string][]byte, and each reader ranged it independently — so HKEYS[i] and HVALS[i] came from two separate randomized iterations, a silent divergence from Redis's pairing guarantee. Fixed with an insertion-order fieldOrder []string on the entry, threaded through HSET/HDEL, AOF replay (free — records replay in order), and BGREWRITEAOF.

The rule: a codec needs a test that composes its two halves. A writer-test and a reader-test that never meet can both stay green while the thing between them is broken. Round-trip your own inverse operations.

The last skip was the most on-the-nose. Every section of TEST_CASES.md passed except §5 (redis-cli compat), which skipped — no redis-cli on the box. TestRedisCLI_ByteCompat does the honest thing (exec.LookPath, t.Skip if absent), but "green because it skipped" is the exact blind spot. Rather than weaken the test, I gave it the tooling: a scripts/redis-cli shim wrapping docker run --rm --network host … redis-cli "$@" (the benchmark already runs valkey/valkey:8-alpine, which ships a redis-cli symlink), plus make compat prepending scripts/ to PATH so the test's own LookPath resolves to Docker. No test change, no native dependency, all 20 cases pass — the PATH-shim-over-LookPath seam un-skips any "needs an external binary" test.

Honourable mentions — three smaller bugs that earned a line

The teatest flake that was a wrong assertion. The TUI paging smoke first asserted "after ], page-1 key k000 is gone from the view." It passed alone and failed under the full -race suite. Not timing — teatest's Output() is the cumulative frame stream, not the current screen, so once k000 was ever drawn, !contains("k000") can never hold again. Replaced with a positive assertion on k050 (which lives only in page 2's SCAN range). Assert on what appears in a cumulative stream, never on what should have vanished.
The PING whitelist the readiness probe depended on. Auth whitelists {AUTH, HELLO, PING} pre-auth — a deliberate deviation from Redis, which gates PING. I'd written it down as an operability-over-parity call, and then the e2e harness's own readiness probe turned out to depend on it: it PINGs before the test can AUTH. The deviation argued for itself — a liveness check shouldn't need credentials.
go mod tidy prunes deps until first import. The observability work fetched the whole OTel module set up front; the first tidy dropped everything not yet imported. Rather than fight it, I let each commit's first import re-add its module — so the go.mod delta in each commit is exactly what that commit uses. A cleaner history fell out of not fighting the tool.

Proof it composes

Rules and war stories are claims. Three artifacts back them.

Byte-compat is mechanical. The dual-protocol sweep replays every command over a live socket on both protocols and asserts the exact bytes. Because the reader is RESP2-only, RESP3 replies are asserted as raw bytes via io.ReadFull — which is more honest for a compat sweep than round-tripping through a parser. A live probe against the built binary, one connection, HELLO 3 then GET nope then PING:

%7\r\n$6\r\nserver\r\n$5\r\ntoykv\r\n … $5\r\nproto\r\n:3\r\n$2\r\nid\r\n:1\r\n …
_\r\n
+PONG\r\n

Native % map on HELLO 3, native _ null on the miss, +PONG unchanged — and the RESP2 client gets $-1 for that same miss, byte-identical to v1.

Durability holds with tracing compiled in. The crash suite and the test/chaos soak stay green with spans wrapping the AOF path (rule 4) — the latency signal survives, the ordering is untouched. And off-by-default is measured, not assumed: TestObserveCommand_Disabled_AllocBudget pins the disabled path at ≤16 allocs/op, turning "OTel-off is free" from an assumption I'd got wrong into a gate-enforced benchmark (rule 6).

Decision records as crystallization — the supersede trail

Both earlier posts landed on the same discipline: write the decision record the moment the decision is forced by code, so it records what the code already decided. toymq v2 added the version-2 wrinkle: most v2 records retire an earlier one, and the honest move is to write a new record with a Supersedes header rather than quietly editing the old one to look prescient. toykv v2 has the same trail — but it also has the sharpest example yet of why the discipline matters, because one of its records was amended by a bug found after release.

The tagged-union store's decision record originally said the hash iterated the Go map independently, "leaving order unspecified." When the hash-order divergence surfaced from writing the test guide, I didn't delete that sentence. I amended the consequences with the correction, dated, naming the fix branch: "Corrected post-v2.0.0 (fix/resp3-reader-hash-order): the entry now tracks field insertion order … so HKEYS[i]↔HVALS[i]↔HGETALL correspond as Redis guarantees." The record now carries its own mistake, legibly, next to the fix. A future reader asking "why is there a fieldOrder slice?" gets the whole timeline — the wrong assumption, the bug, the fix — instead of a clean story that pretends the divergence never shipped.

The same thing happened to the observability record: its post-release amendment records that "no-op providers make it free" was false, keeps the original claim visible, and appends the measurement that disproved it plus the memoization that fixed it. Two records that carry their own corrections.

The rule: supersede and amend, don't overwrite. A decision record you edited to look right is a decision with no history — and the most valuable thing a version-2 decision log can teach is when you were wrong and how you found out.

What I'd do differently

The codec round-trip test should have shipped with the codec. The moment you have an encoder and a decoder, the first test is decode(encode(x)) == x — cheaper than every golden I wrote while HELLO 3 stayed broken under our own client.
The parity A/B belonged in the observability work, not the release pass. The benchmark existed the whole time; a gate you re-read instead of re-run is theatre. "Run the A/B" should have been an exit criterion of the work that introduced the risk.
The security review should have been scoped one layer wider from the start. It found the pre-auth DoS because it looked below auth — but that was luck of framing. "What runs before the gate you're auditing?" should be step one of any auth review.
Verification is a phase, not a suffix on "polish." Folding it into the release is why two bugs almost shipped and two more shipped anyway. Budget it as its own phase — and treat writing the manual test guide as a run, because it is one.

What's next

v2 makes toykv deployable single-node: safe by default, byte-compatible, observable. Going multi-node stays open on purpose — like toymq's v3 question, the roadmap keeps three honest trajectories: A stop at v1.x; B v1 → v2, usable single-node, stop here; C go distributed via ToyRaft, only if it ships as a vendorable library. The default is B, reviewed after each major tag — and the SCAN seq-cursor and atomic keyspace ops are already shaped so a future HSCAN/SSCAN or Raft-applied op fits without a rewrite, keeping the option cheap to exercise and cheap to decline.

The nearer work is the backlog the decision records are honest about: TLS in the shared client (the TUI and CLI still can't reach a TLS-only server — a deliberate deferral), rate-limiting on AUTH, and env/file sourcing for -requirepass so it stops showing up in ps. None of them earn a major. All of them are additive. That's the point.

The one-line version

The series, in four rules that are the same rule. toymq v1: zero values are not sentinels. toykv v1: durations are not deadlines. toymq v2: derived state is not a source of truth. toykv v2: a green check is not a verified one. Each is a thing that looked authoritative until something — a restart, a bind, a first real run — called its bluff. Part 1 said write the bytes you'd send, and stamp them with a version. Part 2 says you earn a major by changing one contract, not by adding seven features — and the roadmap that calls all seven done is recording what you intended, not what you ran.

Source: prajwalmahajan101/toykv. The v2 arc in detail: docs/ROADMAP.md, CHANGELOG.md, docs/TEST_CASES.md, and the seven v2 decision records in docs/adr/ — each captures one decision above, and two carry their own post-hoc corrections. Corrections welcome — open a discussion or file an issue.*

Companion posts in this series: Building toykv (part 1) — the durable core this one builds on — and Building toymq v2 — the sibling project that earned its major by protecting a different contract.

Building toymq v2: eight features, one durability contract, and the bug that proved it held

Prajwal Mahajan — Mon, 13 Jul 2026 08:50:47 +0000

A retrospective on the v1 → v2 arc of toymq — the jump from a single-node persistent message broker that works to one you can actually deploy. The first post was about building the dangerous parts first. This one is about adding features to a durable system without breaking the durability.

TL;DR — the decision, and the bug, you should remember

Part 1 built the bottom of the stack first: WAL → protocol → broker. The thesis was build the thing that, if it's wrong, forces a rewrite of everything above it. v1.0 shipped a broker that survives SIGKILL with zero acked-message loss.

v2 is a different problem. The dangerous part is already built and correct. The job now is to bolt on eight features people actually need — auth, TLS, partitions, backpressure, retention, DLQs, delayed messages, trace correlation — without regressing the one property v1 spent four days earning. Half of them touch the WAL or the wire format, and any one can quietly reintroduce a duplicate, a lost ack, or a torn recovery.

The decision to remember is the smallest one. v2 M1 needed the dedupe index to survive a restart. The obvious move — the move the roadmap originally wrote down — was a sidecar dedupe.json, debounced and atomically swapped, exactly like the offsets file already does. I threw it out before writing a line of it (the rebuild landed in e943980). A sidecar is a lossy cache of a truth the WAL already holds: it lags by its debounce window, so a crash between a dedupe insert and the next flush drops the key and re-materialises the exact duplicate it was supposed to prevent. The right answer was to rebuild the index from the WAL on Open — the scan already visits every record, via a WithRecoveryVisitor hook the broker funnels each recovered record through. Zero new files, zero new fsync path, zero gap.

That collapses to a one-liner that belongs to a family. The other posts in this series each earned one the same shape:

Post	The hero rule	The trap it names
`toymq` v1	zero values are not sentinels	`lastAcked == 0` meant both "acked id 0" and "never acked"
`toykv`	durations are not deadlines	`EX 5` on disk re-applies against `now` after every restart
`toymq` v2 (this post)	derived state is not a source of truth	a `dedupe.json` sidecar re-materialises the duplicate it was meant to kill

The rule: when you're about to persist derived state, ask whether the source of truth already persists it. Usually it does, and the sidecar is a second consistency problem wearing a helper's clothes.

All three are the same bug wearing different clothes: a value that looks authoritative until the system restarts in a state it didn't anticipate. v1 built the dangerous part first; toykv took the opposite call three times on a sibling project. This post turns the axis again — same project, one year of design decisions later — and the question is no longer "how do I get durability right?" but "how do I keep it while everything around it changes?"

And the bug to remember is the one where I almost didn't. In M6, two features I'd built and tested separately — segment retention and delayed messages — turned out to quietly corrupt each other: the retention sweeper would evict an old WAL segment that still held a DELAYed message scheduled to fire in the future, silently deleting an acked publish forty minutes before it was due. Not a crash. Not a race. A message lost to a feature. That's the whole thesis in one bug: once you've earned a durability contract, the thing most likely to break it isn't hardware — it's the next feature. The full story is in the M6 section; the rule it earned runs through the whole post.

The numbers: ~17.6k lines of Go (up from ~10k), 27 ADRs (up from 17), 53 test modules, four binaries, 110 commits across eight milestones (M1–M8), two deliberate wire breaks clustered behind one major bump, and a cross-product integration matrix — {per-message, batched} × {plain, TLS} × {auth, no-auth} × {1, 4 partitions} — that has to stay green before the tag moves.

The order:

M1 dedupe-persist → M2 batched-fsync → M3 HELLO/auth/TLS → M4 partitions
   → M5 backpressure → M6 retention/DLQ/delay → M7 correlated telemetry
   → M7.5 Grafana stack → M8 integration matrix + tag v2.0.0

The point of this post, like the last one, is the order and the rules — not the code.

In v1 the rule was "risk sinks down — build the bottom first." This diagram is that rule rotated ninety degrees: with the bottom already frozen, risk drains left-to-right, and the two red WIRE BREAK nodes sit adjacent on purpose so the one major bump pays for both.

Why the order is the design

v1's ordering rule was "risk sinks down; build the bottom first." v2's is subtler, because everything sits on top of a bottom that's already frozen. Three principles did the work.

Correctness gaps land before performance wins. M1 (dedupe persistence) closes a durability hole; M2 (batched fsync) is a throughput win. M1 ships first, because a performance win layered on a correctness gap just loses data faster. Spend the correctness budget before the speed budget.

Lift the deployment ceiling before building what needs it lifted. M3 (auth + TLS) makes the broker safely reachable off-host; M4 (partitions) scales it. Partitions before auth would ship a multi-partition broker nobody can run off localhost — scaling a thing you can't deploy.

Cluster the wire breaks behind one major bump. v2 has exactly two: the HELLO handshake (M3) and the partition arity on PUB/MSG/ACK/NACK (M4). Everything else — batched fsync, PAUSE/RESUME, retention, DLQ, DELAY, TRACEPARENT — is strictly additive, so a pre-v2 client is processed byte-for-byte as before.

The rule: you get one wire break per major version. Spend it on everything at once, gated behind the same bump.

Each milestone also owns its own crash/concurrency/wire-compat tests. There is no catch-all "integration pass at the end" that discovers M3's bug during M7's work. The risk test lives in the milestone that introduces the risk.

Five rules v2 earned

Like part 1, I didn't write these up front. I extracted them by reading the ADRs and asking what did I refuse to compromise on?

1. The durability contract is sacred. A feature negotiates around it, never through it.

M2's batched fsync coalesces many appends into one fsync. The temptation is to let committed advance when bytes hit the write buffer. It doesn't — committed still advances only after the fsync returns, exactly as in v1. Batching changes when the fsync happens, never what an OK promises. The one genuine weakening, --fsync=none (page-cache only), is opt-in, never the default, and documented loudly. If a feature has to weaken durability, make the weakening explicit and impossible to hit by accident. Retention (M6) is the same rule from the other side — the M6 war story below — where a sweeper almost deleted an acked message a feature had promised to deliver.

2. Every feature is designed around the seam the distributed future needs.

This is what makes v2 more than a feature dump. v3 is toyraft — multi-node replication where the Raft log is the source of truth and each node rebuilds local state by Apply-ing committed entries. That model reached back and shaped v2's single-node code:

Dedupe rebuild-from-WAL (M1) is exactly the raft.StateMachine.Restore seam v3 needs — a follower must rebuild dedupe from the log, never from a per-node file Raft never replicated. The sidecar would have been torn out; the WAL-rebuild seam is reused verbatim.
A partition (M4) is the unit v3 wraps a Raft group around. Topic became a thin router over a Partition type that owns the WAL, dedupe, consumers, and offsets — not only for single-node parallelism, but so v3 gets one Raft group per (topic, partition) without re-architecting.
The determinism debt is written down where it'll be paid. topic.go's time.Now() and local MsgID counter must move to the proposer under Raft (where Apply must be byte-identical on every node) — recorded as v3 M1's first task, not silently deferred.

The rule: when you know where the project is going, let the destination break ties between designs that look equal today. The one that becomes a seam beats the one that becomes a rewrite.

3. Breaking an old client must fail safe or opt in — never silently misbehave.

Both of v2's wire breaks have an escape hatch that's a design choice, not a courtesy: HELLO ships with a --require-hello=false migration window (M3), and 1-partition topics keep the pre-M4 on-disk layout byte-for-byte, so a v1 data directory recovers unchanged (M4). Both are detailed below. Additive features follow the same discipline — TRACEPARENT (M7) is an optional line, and a malformed one degrades to a fresh root span rather than erroring the connection. The parser stays dumb; the W3C spec stays the single source of truth.

4. Backpressure is about the slow half you aren't looking at.

The obvious bug and the real bug differ. Delivery did have backpressure — against a client that stops reading (the send channel and TCP buffer fill). The real gap is a client that reads eagerly and acks slowly: the buffers drain, delivery keeps producing, and inflight grows unbounded. The fix ties delivery rate to the ack rate, not socket readability (the credit window in M5 below). The rule generalises: when you add backpressure, find the consumer that's fast at the thing you're measuring and slow at the thing you aren't.

5. Observability that needs its own dependency in the hot path is the wrong observability.

v1's rule was "observability does not change behavior." v2 extends it: it doesn't get to change dependencies either. M7 needed W3C trace propagation on the wire, but pkg/client is deliberately stdlib-only — so rather than drag OpenTelemetry into every consumer's dependency closure to carry one header, propagation is a callback (client.WithTraceparentFunc) the caller wires to their own otel context. The header crosses the wire; the dependency doesn't cross the boundary.

The dangerous parts of v2, one milestone at a time

M3 — the handshake response that got dropped 40% of the time

The HELLO handshake looked trivial: read the first line, validate, reply HELLO 1 OK or ERR AUTH and close. The first design routed that reply through the session's normal response channel — the same channel the async writer goroutine drains. On a rejection, close(quit) raced the queued ERR in the writer's select, and the error lost about 40% of the time under load. The client got an EOF instead of a reason: "connection closed" instead of "bad token."

The fix is structural, not a lock. Do the handshake synchronously, inline, before the async writer goroutine exists (ca2b7cd): write the response straight to the buffered writer — there's no channel to race and no quit to lose to, because the concurrency that caused the race hasn't been spawned yet. The client mirrors it. Auth itself is boring on purpose — bearer tokens compared with crypto/subtle.ConstantTimeCompare, never logged.

The rule: a response-delivery race is best solved by not having the response-delivery machinery yet. If a message must not be lost, send it before you spawn the goroutines that could lose it.

M4 — Topic becomes a router, and the flat layout survives

Partitions could have been a rewrite. They weren't, because the publish/deliver/ack/offset logic didn't change — it got re-scoped. Everything the old Topic owned (WAL, dedupe, consumers, pubMu, offsets) moved to a Partition; Topic became a thin round-robin router over []*Partition. MsgID stays monotonic per partition — total order within a partition, none across — so a (partition, msgID) pair is the message identity now.

Routing is layered: explicit <topic>#<n> wins; else the routing key hashes (fnv1a % N); else keyless publishes round-robin. Because wal.Open was already per-directory, the WAL needed no change — each partition points it at its own subdir. And back-compat falls out of one decision: meta.json is written only when N > 1, so its absence is the "flat, pre-M4 layout" signal. The two shapes on disk:

data/topics/orders/          # N > 1: partitioned
├── meta.json                #   {"partitions": 4}   ← the only new artifact
├── 0/ 000000.log  offsets.json
├── 1/ 000000.log  offsets.json
└── ...
data/topics/logs/            # N == 1 (or any pre-M4 dir): FLAT, unchanged
├── 000000.log               #   no meta.json — byte-for-byte the v1 layout
└── offsets.json

// topicPartitionCount: meta.json when present, else 1 for a flat/legacy
// layout — so a pre-M4 data dir recovers unchanged as a 1-partition topic.
data, err := os.ReadFile(filepath.Join(topicDir, "meta.json"))
if err == nil {
    var m topicMeta
    _ = json.Unmarshal(data, &m)
    return m.Partitions, true, nil          // N > 1: partitioned
}
if os.IsNotExist(err) {
    // No meta.json: a flat/legacy 1-partition topic if the dir exists.
    if _, statErr := os.Stat(topicDir); statErr == nil {
        return 1, true, nil                 // byte-for-byte the old layout
    }
}

M5 — a wake channel, because `sync.Cond` can't say no

The mechanism (rule 4's fix) has a sharp edge worth keeping. The credit gate must honour context cancellation — a subscription teardown has to unblock a goroutine parked waiting for credit. sync.Cond is the textbook "block until a condition changes" tool, but you can't cancel a cond.Wait() from a select/ctx. So the gate is a buffered-1 coalescing channel plus a re-check loop (022eaea), with the receive in a select alongside ctx.Done():

// awaitCredit blocks until this consumer may deliver one more message —
// not paused and fewer than window inflight — or ctx is cancelled.
func (c *Consumer) awaitCredit(ctx context.Context) bool {
    for {
        c.mu.Lock()
        ready := !c.paused && len(c.inflight) < c.window
        c.mu.Unlock()
        if ready {
            return true
        }
        select {
        case <-c.wake:      // Ack/RESUME nudged us — loop and re-check.
        case <-ctx.Done():  // subscription torn down — the exit sync.Cond can't do.
            return false
        }
    }
}

PAUSE/RESUME rode along as additive session-scoped verbs — with one honest caveat the flow-control ADR spells out: PAUSE is best-effort to within one in-flight message, because a goroutine already parked in the WAL read will still deliver the one record it was awaiting. Cooperative flow control, not a hard barrier — and said so.

The rule: sync.Cond blocks but can't be cancelled. When a blocked goroutine has to honour ctx, reach for a coalescing channel and a re-check loop, not a condition variable.

The gate was right the first time; the test was the war story. My first "does PAUSE actually stop delivery?" test asserted "no message arrives while paused" — and hung for the full 600-second timeout. I went hunting for a deadlock in the gate. There wasn't one: the test never acked what it received, so the 100ms redelivery ticker re-pushed the same messages forever, and the drain loop's idle timer never got the quiet gap it fires on. The gate did exactly what I asked; the test asked a question that could never come true. The fix set the window to 1 so the goroutine parks in the gate (zero PAUSE slip) and the drain gets its silence. A test that hangs isn't always a deadlock in the code — sometimes it's a liveness bug in the test's own premise, and the redelivery ticker I forgot to account for was the tell.

M6 — three features that had to not corrupt each other

Retention, DLQ, and delayed messages shipped as three stacked PRs, and their interactions were the whole game.

Retention turns the single WAL file into rolling numbered segments and drops whole sealed segments past a size/duration bound (the active segment is never touched). A resuming consumer whose offset fell below the retained floor gets an explicit ERR OUT_OF_RANGE — not a silent skip, not a crash. Segment boundaries are also the natural v3/toyraft snapshot reference (rule 2 again).
DLQ republishes a message that fails N times onto an auto-created <topic>.dlq, via a deterministic dlqMove seam. .dlq topics are loop-guarded so a poison message can't ping-pong. In v3 the leader proposes the move; the seam is already shaped for it.
Delayed messages stamp an append-only VisibleAtNs at propose time; the delivery goroutine parks until it fires, preserving per-partition order. It's persisted across restart for free because it lives in the WAL — and, per rule 1, retention learned not to evict a segment holding an un-fired delayed record.

That last clause is the near-miss from the intro (8d0c9dd). Retention drops the oldest sealed segments — and a delayed message scheduled for the future lives in an old segment (published long ago, fires later). Composed naïvely, the sweeper evicts a message the broker had promised to deliver. The fix makes the evictable floor the minimum of the retention bound and the oldest un-fired VisibleAtNs. Two features that each looked correct alone would have corrupted each other; the invariant — retention respects delivery — is the thing that had to be designed, not either feature.

M7 — the continuation that the parser refused

The plan was full producer→consumer trace stitching. It hit a wall that was correct to exist. Continuing the trace onto the delivered MSG frame meant a sixth token on a line the client's parser requires to be exactly five fields — an instant break for every pre-M7 client. And genuine stitching across a durable log needs the producer's trace context persisted in the WAL record (delivery can be minutes later, or post-restart) — a record-format change with all the determinism weight of the dedupe-recovery decision. So I scoped the continuation out, shipped the tested high-value half (producer→broker), and recorded the deferral in the traceparent ADR. Sometimes the right move is to defer a wire change loudly, reason on record, rather than force a break to look complete.

M7 also left a footgun I'll name so the next person doesn't rediscover it: log↔trace correlation only fires on slog.*Context calls, because the handler reads the span off the record's context. A bare slog.Info in a traced path silently drops the trace_id. There's a test that asserts the join to catch regressions, but the real fix would be a lint rule: traced paths log with *Context.

Proof it composes

Rules and war stories are claims. Three artifacts back them.

The M8 matrix — every feature crossed with every other. The v2 features aren't independent: batched fsync must survive TLS framing, partitions must fan in under auth. So M8's owned risk test is the full cross-product {per-message, batched} × {plain, TLS} × {auth, no-auth} × {1, 4 partitions} — 16 cells, each fanning keyless publishes back in over <topic>#* and asserting exactly-once delivery plus per-partition MsgID monotonicity. Verbatim, under the race detector:

$ go test ./internal/integration/ -run TestM8CrossProductMatrix -race -v
=== RUN   TestM8CrossProductMatrix/per-message/plain/no-auth/p1
=== RUN   TestM8CrossProductMatrix/per-message/tls/auth/p4
=== RUN   TestM8CrossProductMatrix/batched/plain/no-auth/p4
=== RUN   TestM8CrossProductMatrix/batched/tls/auth/p1
    ... (16 cells: 2×2×2×2)
--- PASS: TestM8CrossProductMatrix (0.00s)
ok      github.com/prajwalmahajan101/toymq/internal/integration   1.185s

The durability cost is real and honest. Per-message durability is measured, not hand-waved. On commodity NVMe, toymq-bench with 8 producers pushing 20k×128-byte messages under the default --fsync=per-message:

$ toymq-bench --producers 8 --msgs 20000 --size 128 --fsync per-message
throughput  45162.6 msg/s   5.51 MiB/s
latency     min=27µs  p50=149µs  p95=349µs  p99=593µs  max=2.2ms

Every one of those 45k/s publishes crossed an fsync before its OK — the rule-1 contract, priced. --partitions 4 holds the aggregate (~11.4k/s per partition) because each partition is an independent log. The number that matters isn't the throughput; it's that it's honest about what it bought.

And the contract holds in the new fsync mode under crash. The chaos harness runs the SIGKILL soak in batched mode too — a producer publishing through three SIGKILL/restart cycles, a consumer recording every id, the invariant being that every acked publish is eventually delivered:

$ CHAOS_FSYNC=batched CHAOS_DURATION=100s go test -tags chaos ./test/chaos/ -v
chaos: consumer uniqueSeen=10560 totalDeliveries=10584 errors=6 resubs=4
chaos: supervisor restarts=3
--- PASS: TestChaosSurvivesSIGKILL (100.26s)

3 restarts, zero acked-message loss. totalDeliveries exceeds uniqueSeen by exactly the 24 at-least-once duplicates the contract permits; the 6 errors are publishes that never got an OK because the broker was dead mid-kill, so they were never owed durability. Same guarantee v1 proved with per-message fsync — now with group commit on. The feature negotiated around the contract; it didn't go through it.

ADRs as crystallization — the supersede trail

Both earlier posts landed on the same ADR discipline: write it the moment the decision is forced by code, so it records what the code already decided. v2 adds the wrinkle only a version-2 project can teach. v1's ADRs recorded first decisions; most of v2's retire an earlier one — batched fsync supersedes per-message-only, partitions supersede the single-log frame, traceparent supersedes the "no trace context" limitation. The temptation with each was to open the old ADR and quietly edit it, making past-me look like he'd planned for this. I didn't: the old ADR stays untouched, and the new one carries a Supersedes header explaining what changed and why now. The log stays a truthful timeline of when I changed my mind, not a re-touched photo.

That matters for the reason the durability contract matters. The contract is sacred; the decisions around it evolve — but only in the open. A superseded ADR has a visible history; an overwritten one pretends it never had an alternative. When v3 asks "why rebuild dedupe from the WAL, not a sidecar?", the answer isn't folklore — it's an ADR with the sidecar sketch it beat still legible inside it.

The rule: supersede, don't overwrite. An ADR you edited to look right is a decision with no history — and a decision with no history is indistinguishable from luck.

What I'd do differently

The MSG-continuation dead-end was checkable at design time. One grep at the client's MSG parser would have shown the fixed arity before the spec promised a continuation. Check the strictest consumer of a format before you promise to extend it.
The *Context-only correlation constraint wants a lint, not a test. A test catches the regression after you write it; a lint stops you writing it. Silent-miss footguns deserve mechanical enforcement.
The PAUSE/redelivery interaction should have been in the brainstorm. Both flaky M5 test drafts trace to not accounting for the redelivery ticker up front. "How does this feature interact with redelivery?" belonged in the design gate.
Stand the full LGTM stack up before calling M7.5 done. The configs are statically validated (docker compose config, promtool check rules, a metric cross-check) — strong, but a live up --build click-through is the difference between "should pivot" and "does pivot." Static validation is necessary, not sufficient, for a UI.

What's next

v2 makes toymq useful single-node. v3 is where it earns the "distributed broker" framing, gated on a real dependency — toyraft — existing and being embeddable. The roadmap is honest that three of toyraft's capabilities (snapshots/compaction, membership changes, ReadIndex reads) are stubbed today, so parts of v3 are upstream-blocked until they land.

And it keeps one decision open: A — stop at v1.x; B — v1 → v2, usable single-node, stop here; C — go distributed, only if toyraft is real. The default is A, reviewed after each major tag. Shipping v2 doesn't commit me to v3 — it just makes v3 buildable if the dependency shows up. That's the whole point of designing every v2 seam around where v3 would need it: the option stays cheap to exercise and cheap to decline.

The one-line version

The series, in three rules that are the same rule: v1 — zero values are not sentinels. toykv — durations are not deadlines. v2 — derived state is not a source of truth. Each is a value that lied about being authoritative until a restart called its bluff. Part 1 said the storage is the project. Part 2 says the durability contract is the project — and once it's earned, every feature negotiates around it, never through it.

Source: prajwalmahajan101/toymq. The v2 arc in detail: ROADMAP.md, CHANGELOG.md, and the ten v2 ADRs (0018–0027) in docs/adr/ — each records one decision above, with the alternatives it beat and the ADR it supersedes. Part 1: Building toymq.

Companion posts in this series: Building toymq (part 1) — the durable core this one builds on — and Building toykv — the sibling project that took the opposite call three times.

Building toykv: a from-scratch persistent KV in Go, and why I took the opposite call from toymq three times

Prajwal Mahajan — Wed, 17 Jun 2026 16:39:53 +0000

A retrospective on toykv — a single-node, in-memory key-value store with append-only-file persistence, RESP2-compatible on the wire, three binaries, and a v1.0.0 I shipped after about four weeks of work.

TL;DR — and the rule I want you to remember

Some rules generalise across systems by inversion. toymq taught me zero values are not sentinels. toykv taught me durations are not deadlines. Same shape of bug, opposite axis — and along the way I ended up answering three load-bearing questions in the exact opposite direction from toymq:

I shipped toymq with no version byte on its WAL. toykv's AOF starts with one.
I gave toymq its own wire protocol and a separate WAL record frame. In toykv I write the same RESP arrays I speak on the wire straight into the AOF.
toymq had its own delivery semantics to defend. For toykv I chose RESP2 specifically so redis-cli and go-redis/v9 become free third-party test harnesses.

The proof those opposite answers were right is the crash matrix — nine rows, each pinned by a test file that exists today, each test owned by the part of the system that introduced the risk. Every row is cheap because of one of those three decisions.

The hero rule of toykv — sibling of toymq's lastAcked uint64 == 0 zero-value-sentinel bug, reached from the other direction — is one line:

Absolute deadlines, never relative durations.

Store EX 5 on the wire; write PEXPIREAT 1718659200000 to disk. toymq taught me that zero values are not sentinels. toykv taught me that durations are not deadlines. Same shape of bug, opposite axis.

The numbers: ~5k LOC of Go, 6 ADRs (I budgeted four), 15 journal entries, three binaries (toykv, toykv-cli, toykv-tui), v1.0.0 tagged on 2026-06-17.

What toykv is, and isn't

A learning artifact. Single-node, in-memory, no auth, no TLS, strings only. 18 commands from the Redis surface — GET, SET (with NX/XX/EX/PX/EXAT/PXAT), DEL, INCR/DECR, KEYS, FLUSHDB, EXPIRE/TTL/PEXPIRE/PEXPIREAT/PERSIST, BGREWRITEAOF, and a handful of metadata commands. Under appendfsync=always (the default), throughput tops out in the low thousands of SETs per second on commodity NVMe — a per-write fsync is the durability commit point and the cost is honest. If you need a real KV store, reach for Redis or Valkey. If you want to understand what "durable" means in one, build one.

The crash matrix

I lay it out the way I read it: every row of fault surface has exactly one test file, and the test lives next to the layer that introduced the risk. Composed-fault tests sit on their own at the bottom because composing faults is a different job from proving any one of them.

Fault surface	Risk	Test file	Invariant proven
AOF append + replay	acked SET / DEL lost on SIGKILL under `fsync=always`	`internal/server/aof_crash_test.go`	Every record acknowledged before the kill replays on restart
Partial-tail handling	crash mid-record corrupts replay state	`internal/aof/replayer_test.go`	Replay rejects a torn record; offset reported, server refuses to serve until truncated
TTL lock-upgrade race	sweeper drops an unexpired key under load	`internal/store/sweeper_test.go`	N writers × 1 Hz sweeper → zero spurious `(nil)` for unexpired keys
TTL crash round-trip	TTL records lost across SIGKILL + restart	`internal/server/aof_ttl_crash_test.go`	v2 replay accepts v1 records; expiry decodes round-trip
`BGREWRITEAOF` during writes	rewrite races a concurrent SET; tail loses data	`internal/aof/rewriter_test.go`	Side-buffer captures all live appends; rewrite + tail merge with no loss
Crash during rewrite	partial `.aof.tmp` left, no canonical AOF survives	`internal/server/aof_rewrite_crash_test.go`	Exactly one of `{old, new}` present after restart; replay consistent
Shipped binary protocol drift	`redis-cli` / `go-redis` regression invisible to in-process tests	`test/e2e/protocol_*_test.go`	Byte-compat for every shipped command
`BGREWRITEAOF` + restart	restart after compaction loses data	`test/e2e/rewrite_restart_test.go`	Mixed-workload state survives rewrite + restart
Composed faults (kill + pause + rewrite + writes)	failure modes only visible when multiple faults overlap	`test/chaos/invariants_test.go`	Acked-SET survival, monotonic INCR, no panic across soak

The protocol-compatibility layer (rows 7–8) deliberately introduces zero new crash invariants. It verifies that the bytes leaving the socket match the spec third-party clients expect; everything below the socket has already been proven elsewhere. The chaos suite (row 9) is the only layer that composes faults. Composed-fault tests as the primary durability proof are slow and flaky; composed-fault tests as a release-confidence soak after each component is independently proven are exactly what you want.

The thesis I want you to take away: every row is cheap because of one of the three decisions below. Where a crash matrix becomes expensive — where the test file is 800 lines, or two rows have to share state — that's usually the symptom of an architecture decision made elsewhere.

The first three layers introduce new fault surface, each proven in isolation by a dedicated crash test. The protocol layer verifies the shipped binary speaks the spec; it adds no new crash invariants. The chaos layer composes everything underneath into one soak. That shape is the proof the architecture is paying its rent.

Inversion 1 — a version byte you'll thank yourself for, and absolute deadlines

toymq's WAL has no version byte. The argument I wrote in that retrospective is good: defending the wrong byte forever is worse than the one-line migration of adding one later.

toykv's AOF starts with eight bytes: "TOYKV\x00\x00" plus a one-byte version. The reasoning is also good, for a different shape of problem: the AOF outlives the binary that wrote it. If I run toykv on a tempdir on Monday and the same dir with a newer binary on Friday, the Friday binary has to know what assumption the Monday bytes were committed under. There is no person to ask. The byte is the answer.

I made the call before I wrote a line of replay code, and wrote it up as an ADR alongside the first AOF commit. It paid for itself inside four weeks: I needed a new on-disk shape for TTL, and the version byte let v2 binaries accept v1 files transparently. The replay loop is the same eight-line dispatch table; v2 just knows about a PEXPIREAT token v1 didn't.

The rule: the cheaper migration is the one your future self can opt into, not the one their replay code is forced into. A version byte you don't use is a no-op. A version byte you didn't write is a migration script.

Absolute deadlines, never relative durations

The companion ADR made the second call: every TTL command I accept on the wire rewrites to absolute PEXPIREAT <unix-ms> before it touches the disk. A wire SET k v EX 5 becomes an AOF SET k v + PEXPIREAT k <now-ms + 5000>. The wire stays ergonomic; the disk stays unambiguous.

Imagine the alternative. Store EX 5 verbatim in the AOF. Crash. Restart twelve hours later. Replay reads EX 5 and applies it against time.Now() — every TTL in the file just got extended by twelve hours. Worse, the bug is silent: keys don't disappear, they linger. The test that catches this is a multi-day fixture, because the only way to notice is to wait past the original expiry. That's the toykv-shaped sibling of toymq's lastAcked == 0 sentinel collision — a value that looks valid until the system restarts in a state it didn't anticipate. Same shape of bug, different axis.

The rule: absolute deadlines, never relative durations. The wire can speak in human-friendly relative time; the disk has to speak in moments. A duration without a reference point isn't a deadline — it's a wish.

This is the rule I earned. Everything else in the post is the architecture I built to make the rule cheap to enforce.

Replay reads PEXPIREAT k 1718659200000. The math works regardless of when replay runs — a year later, the answer is "this key was already expired"; a second after the crash, "this key has 4.999 seconds left." There is no clock to argue with. The wire stays ergonomic; the disk stays unambiguous.

The AOF on disk

The file format is deliberately boring:

data/
└── appendonly.aof    ← header + RESP-encoded record stream

One file. No manifest, no index, no segment rotation in v1. The header is eight bytes; everything after is a stream of RESP arrays. The recovery scan walks the file from offset zero on Open — O(disk), but correct by construction. A manifest would be a second consistency problem on top of the first.

Field	Type	Size	Notes
`magic`	bytes	7	Literal `"TOYKV\x00\x00"` — fails fast on a wrong file
`version`	`u8`	1	`0x01` (strings only) or `0x02` (adds `PEXPIREAT` for TTL)
`record_*`	RESP	variable	Same `*<n>\r\n$<len>\r\n…` frame the wire speaks

There is no per-record CRC. RESP framing already catches truncated bulks and mismatched array counts; adding a CRC for a learning artefact is belt-and-braces I deliberately deferred. The first real-world corruption that slips past RESP-level parsing is the moment to add one — not before.

Inversion 2 — write the bytes you'd send

The journal entry that wrote itself, docs/journal/02-aof.md, has this line:

"The whole AOF format is RESP arrays on disk. Append calls the existing resp.Writer against a buffered file handle; Replay calls the existing resp.Reader. One codec, two consumers."

That's the architectural claim. Everything I did downstream of it is mechanical. The replay path is nine lines of Go: skip the header, read a RESP array, dispatch it, repeat to EOF. During replay s.aof == nil, so the appendIfLive call inside each mutating handler is a silent no-op. The same nine lines handle real traffic and crash recovery.

The AOF-replay invariant — every record acknowledged before the kill replays on restart — collapses to a single ordering question: does the +OK cross the network before or after file.Sync() returns? Under appendfsync=always, the answer is after. The proof is operational: the crash test (internal/server/aof_crash_test.go) writes ~90 SETs against a self-re-exec child server, sends SIGKILL mid-stream, restarts, and checks every +OK against the post-replay state. -count=10 after the first pass found zero flakes.

I drew this diagram in the same shape as the WAL diagram in the toymq retrospective on purpose. A reader who has seen both will notice that the durability ordering is identical — that's the thing the two projects agree on. The codec choice is what's different, and the codec choice is what made the replay path nine lines instead of a hundred.

The rule: write the bytes you'd send. One codec is cheaper than two consistent codecs, and the second codec is where the corruption story you don't want lives.

Inversion 3 — RESP because the harness is free

For toymq I invented a protocol. The defence is sound: the protocol is the wire, the wire is the contract, an invented wire is the moment a project decides it can defend its own boundaries.

For toykv I chose RESP2. RESP2 is a testing strategy disguised as a feature. The reason is the protocol-compatibility layer of the crash matrix: integration tests against the shipped binary using third-party clients — redis-cli for byte-level conformance, go-redis/v9 for the API a real Go consumer would see. That layer shipped without a line of client code I wrote. The clients ship for free, by people who have never heard of toykv. They don't care about the implementation; they care about the wire. That's exactly the property I wanted.

The crash-matrix tie-in is the load-bearing one: the protocol layer owns zero new crash invariants. It owns protocol invariants — TTL -2/-1 sentinels, EXPIRE against a missing key returning 0, the byte-exact framing of *3\r\n$3\r\nSET\r\n… — but every command was already covered by a crash test in the layer that introduced it. The protocol layer is the validation that the bytes leaving the socket match the spec the third-party clients expect; everything below the socket has already been proven.

The rule: protocol compatibility is a testing strategy disguised as a feature. Two independent implementations of the wire shake out the spec. No single-author project has that property by default — you have to inherit it deliberately.

Honourable mention 1 — the dual-write `bufio` bug

The interesting bug of the project was not the rename-plus-dir-fsync-plus-fd-swap dance for BGREWRITEAOF that I'd been bracing for. The invariants I'd written up in the rewrite ADR worked the first time. What broke was the dual-write Append path inside the rewriter: sideBuf.Len() == 0 after what looked like a successful write.

Root cause: resp.Writer wraps its target in a bufio.Writer. The live path constructs resp.NewWriter(outerBufio), and bufio.NewWriter(outerBufio) is smart enough to short-circuit — it returns the existing *bufio.Writer rather than wrapping it again. One buffer, one flush. When Rewriter.BeginRewrite constructed a fresh mirror writer against a *bytes.Buffer, the argument was no longer a *bufio.Writer. So bufio.NewWriter(&scratch) created a brand new inner bufio layer. Two buffers, two flush points. Records were "written" but pinned in an inner bufio nothing else was going to drain before DrainAndSwap ran. Two-line fix in 736bf63.

The rule: any write path with two sinks has two flush points; both must be ordered against the swap. A bufio layer is invisible — the same line of Go behaves differently depending on the static type of its argument. Never construct a fresh wrapper of a swappable type against a non-bufio target without immediately scheduling its flush.

The invariant the rewrite ADR commits to is "the canonical file is durable and consistent at every instant until the rename." A silently-empty side buffer would have meant the rename swapped in a file missing the appends issued during the snapshot. The replay test was the green-light proof that both sides of the dual-write are actually dual-writing.

Honourable mention 2 — the test was wrong, not the code

The TTL sweeper is a 1 Hz goroutine that samples 20 keys under a read lock, then upgrades to a write lock for the eviction pass — sync.RWMutex doesn't support upgrade-in-place, so the upgrade is release-and-reacquire-and-double-check. The race test (internal/store/sweeper_test.go, commit bab6033) failed on its first stress run with four violations in 468k reads.

The bug was in the test, not the store. The test captured t0 := time.Now() before Get, but Get's internal time check happens later — sometimes much later if the scheduler preempts. A Get that correctly returned (nil) because the entry's TTL had elapsed by the time Get internally checked would be flagged as "spurious nil for unexpired key." The fix forced the memory model onto paper: I now capture tAfter after Get (upper-bounds the internal check) and atomic.Load(lastExpireAt) before Get (synchronizes-with the writer's atomic.Store). Then tAfter < loadedExpireAt is a real violation; no false positives possible.

The rule: if a sweeper and a writer can race, the test has to schedule them, not hope. A test whose pass/fail depends on the scheduler's whim is a test that finds the race detector's bug, not your code's.

ADRs as crystallization — and the ADR I almost didn't write

I budgeted four ADRs for v1.0.0 and shipped six. The interesting one is the TUI ADR.

The TUI ADR (Bubble Tea with an injectable Doer) is where the budget actively pushed against the code. The TUI was the first part of the project that took a third-party dep beyond go-redis/v9 in tests, and the choice — bubbletea vs tview vs hand-rolled — was non-trivial. I knew it was non-trivial because every two days, in a different context, I caught myself re-deriving the same argument for the same choice. That's the signal: when the same reasoning costs me a second time, I didn't decide; I waited.

I had two options. Force the choice into one of the existing four ADRs and write a worse ADR. Or break the budget and write the TUI ADR on its own. I broke the budget. The post-mortem is that the budget itself was the wrong abstraction. I had treated ADR count as a proxy for noise, when the actual signal was ADR quality — and a forced-fit ADR is noisier than two clean ones.

The discipline, identical to the one I wrote in the toymq post: write the ADR the moment the decision is forced by code, never before. ADRs written ahead of code force the code to fit the ADR. ADRs written at crystallization record what the code already decided. The reason that discipline matters is that the TUI ADR was a decision I'd already made three times by the day I wrote it down — I just hadn't admitted I'd made it. The ADR is the admission.

The rule: budget the quality bar, not the count. The cost of an ADR I write is one afternoon. The cost of an ADR I didn't write is a future me re-deriving the same argument in three different files. Those costs aren't comparable, and counting ADRs treats them as if they are.

What I'd do differently

Three regrets from the critical path:

Chaos composition should have shipped with the AOF, not at the end. The test/chaos/ soak (kill + pause + rewrite + writes overlapped) felt expensive enough to defer. It wasn't. Two-thirds of the harness reused the self-re-exec pattern I already had for the AOF crash test. Running a composition soak from the start would have caught the dual-write bug at the point the rewriter landed, because the rewriter would have been driven by a scaffolded integration test instead of an isolated unit test.
Store.Snapshot() should have shipped with the AOF, not landed as a refactor PR when the rewriter needed it. Snapshot is a contract boundary — "give me the current state, expired entries already evicted, in a form I can serialize." I deferred it because the AOF didn't use it yet. The refactor PR that extracted it later was harder to review than the BGREWRITEAOF work itself, because it touched nine files mechanically. Shipping the boundary up front would have cost one extra journal entry.
The ADR budget number was the wrong abstraction. See above. Budget the bar, not the count.

What's next

toykv-go — the SDK extraction. The strategy I committed to is: hold the SDK split until the wire format is locked, then extract internal/client, respfmt, and cmdparse into a sibling repo. v1.0.0 of toykv-go tagged the same day as toykv v1.0.0.

toymessenger — the consumer. The reason toykv and toymq exist as separate projects is that there is a third project, an E2E-encrypted chat, that uses both: KV for session state and presence, MQ for the message log. toymessenger is where the two projects' opposite decisions get tested against a single user.

A toykv v2 only if real use justifies it. The roadmap lists candidate features (lists/sets/hashes, AUTH + TLS, RDB snapshots, observability with INFO + /metrics, SCAN, RENAME), but the rule is the same one Redis uses: don't add a feature until somebody is asking for it and willing to live with the trade-off. A single-node KV that survives a v1 cycle without a v2 ask is a single-node KV that earned the cut.

The one-line version

Write the bytes you'd send. Stamp them with a version. The next person to read them is you.

Source: prajwalmahajan101/toykv. Deeper docs: docs/HLD.md, docs/LLD.md, docs/TESTING.md, docs/BENCHMARKS.md. ADRs: docs/adr/. Release notes: docs/release-notes/v1.0.0.md. Corrections welcome — open a discussion or file an issue.

Companion post: Building toymq — the sibling project this one is in dialogue with.

Building resilience-kit: A Python Resilience Kernel Forged in Production

Prajwal Mahajan — Thu, 11 Jun 2026 08:18:08 +0000

TL;DR

resilience-kit is a framework-agnostic, async-first resilience kernel for Python. Circuit breakers, retries with jitter, token-bucket throttles, SSRF guards with DNS pinning, fire-and-forget audit logs, field-level encryption. Pluggable backends (memory, Redis/Valkey, pybreaker). Thin adapters for FastAPI and Django that wire the primitives into each framework's lifecycle and contain zero business logic. One core, 11 ADRs, two adapters.

The interesting part isn't the primitives — most of them are well-trodden ground. The interesting part is the release rule: v0.1.0 only shipped because I had to upgrade two unrelated starter repos to it and score ≥ 8/10 on each migration. The reports got filed. FastAPI scored 8/10. Django scored 9/10. The gate held. Anything below 8 would have blocked the cut.

That rule — "the kit ships iff the kit survives its own migration" — is what shaped the public surface. Every helper in v0.1.0 (bind_to, from_exception, legacy_env_alias, verify_envelope_contract) exists because the first dogfooding round flagged its absence as a blocker. Every doc gap got patched because the migration reports cited the line number it should have been on. This post is how the kit got built, the five axioms underneath it, the gate that gated it, and the bugs it found in itself along the way.

95 source modules, 52 test modules across unit/contract/integration suites, 11 ADRs.

Five Axioms

I didn't draft these up front. I extracted them after the fact by reading the ADRs and asking what did I refuse to compromise on? They're the load-bearing constraints; everything else follows.

1. One package, many extras. Not a forest of micro-packages.

pip install resilience-kit gives you the core. pip install resilience-kit[redis,pybreaker,fastapi] adds the backends and adapter. No micro-libraries until the surface justifies it. One repo, one CHANGELOG, one tag. Modularity is enforced by import discipline (import-linter with layered contracts) and by entry-point provider discovery, not by package boundaries.

Arrows are hard imports (from resilience_kit.X import Y). Dotted lines are entry-point conformance — no import required, just a Protocol-shaped class registered in the extra's pyproject.toml under [project.entry-points."resilience_kit.cache"] (and siblings). The kit's import graph is acyclic by import-linter contract; consumer apps depend only on the adapter layer.

The bet: a single distributable up to ~10k LOC is simpler to release, version, and depend on than a federation. If the surface ever grows past that, splitting into namespace packages later is mechanical and doesn't break the public API.

2. Protocols, not ABCs.

Every swappable interface — cache, breaker, throttle, audit backend, metrics sink — is a typing.Protocol, not an abc.ABC. Third-party backends import zero kit packages at definition time. They expose a class whose shape matches; mypy --strict verifies the structural conformance.

The rule: your code should conform to a shape, not inherit from a library. This is true dependency inversion — the kit doesn't even appear in the import graph of the thing implementing its protocol. Entry points provide the precedence chain at runtime: explicit > env > entry-point > default.

3. Outer breaker, inner retry. Composition order is correctness.

If you compose a circuit breaker and a retry, the breaker must wrap the retry. If you accidentally invert the stack — retry around breaker — the retry loop will hammer an OPEN breaker, defeating the point of the breaker. This isn't a style preference; it's a correctness property.

I learned this the hard way early on. A caller passed retry_on=(Exception,) to the retry decorator. The decorator happily caught ServiceUnavailableError — the exception the breaker raises to say "stop calling me." The fix (commit f6cebdd) made the dangerous composition fail safe by stripping ServiceUnavailableError from the retry's catch list, no matter what the caller passes:

def _filter_retry_on(exceptions):
    """Drop ServiceUnavailableError from a retry_on tuple."""
    return tuple(
        e for e in exceptions
        if not (isinstance(e, type) and issubclass(e, ServiceUnavailableError))
    )

The rule: the dangerous composition must be the one that fails safe. An inverted stack now fails instantly, because the retry loop refuses to catch the one error a breaker is designed to throw.

4. Async-first with explicit sync bridges. Never trust auto-bridging.

Every primitive's async API is primary; sync wrappers exist only where the primitive is idiomatically sync (Django middleware, DRF throttles, management commands). Crucially, the kit does not use asgiref.sync.async_to_sync. That function caches a thread-local event loop to "avoid overhead" — which means it's a hidden global namespace, and any other library that also caches a thread-local loop will collide with it.

Instead, the kit owns both crossings explicitly:

Long-lived background work (recovery monitor, audit dispatcher): the Django adapter spawns a single daemon thread with its own private event loop. The loop drives the monitor and owns the audit queue. atexit drains the queue on the same loop before closing it.
Short-lived sync→async calls (DRF throttles checking Redis): a per-call asyncio.run(). Sub-millisecond overhead. No shared state. No collision risk.

The rule: a bridge between two worlds must own its own crossing. Don't borrow infrastructure from either side.

5. Adapters are dumb wiring. If your adapter has business logic, your primitive is wrong.

resilience_kit.adapters.fastapi and resilience_kit.adapters.django exist solely to wire kit primitives into each framework's lifecycle: ASGI middleware, FastAPI dependencies, Django AppConfig.ready(), DRF throttle classes, management commands. They translate framework-shaped concepts (a Django MIDDLEWARE tuple entry, a FastAPI Depends()) into kit calls. Nothing else.

If an adapter file ever grows past ~300 LOC, the primitive itself is wrong, not the adapter. This is enforced socially (PR review) and architecturally (import-linter prevents adapters from importing each other or from owning state).

The closing line of this post is the punchier statement of this axiom.

The Dogfooding Gate

The most consequential decision in this project wasn't a code decision. It was a process decision:

"v0.1.0 is a clean cut iff both boilerplate reports score ≥ 8/10."
— docs/RELEASE-PLAN.md §4 verification

I write FastAPI and Django services. Before the kit existed, both starter repos owned the same resilience layer twice — circuit breakers, retries, throttles, SSRF guards, audit logs — copy-pasted and lightly diverged. The kit is the deduplication. But deduplication is easy to talk about and hard to validate; "the kit can replace the boilerplate code" is a claim, not a proof.

So I made it a release gate. Cut a release candidate. Upgrade both boilerplates against it. File a structured report on each migration — blockers, helpers used, missing surface, pain points, doc gaps, ROADMAP suggestions — and a 1–10 outcome score. If either score is < 8, the kit isn't done. Iterate. Re-test.

That cycle ran from 0.1.0rc1 → 0.1.0. Both reports passed:

Repo	Score	Date
`fastapi_boilerplate`	8 / 10	2026-06-10
`django_boilerplate`	9 / 10	2026-06-11

Both reports applied all four primary helpers that shipped in v0.1.0: bind_to(target) for ContextVar mirroring, from_exception(exc) for the kit-shape → adopter-envelope projection, legacy_env_alias() for pre-kit env-var translation, and verify_envelope_contract() for an adopter-side test that the bridge maps every kit exception cleanly. The fact that both reports used the same four — first try, no fallbacks — was the convergent signal that the helper surface was right.

What's interesting isn't that the gate passed. It's what the gate produced.

What the Migration Reports Found

The reports were structured by design — same template, same sections — so the findings could be aligned and compared. Four findings showed up in both:

#	Cross-cutting finding	Target
X1	The public exception bridge lives at `resilience_kit.adapters._envelope.from_exception` — the leading underscore reads as private. Re-export it without the underscore, or rename the module.	v0.1.1
X2	`verify_envelope_contract` raises a flat `AssertionError`. Pytest only surfaces the first failure; programmatic CI dashboards can't introspect a structured result. Return an `EnvelopeContractResult` (list of `(exc_class, ok, reason)`).	v0.1.1
X3	`legacy_env_alias(aliases=...)` replaces the default table. Adopters who want to extend with project-specific aliases must copy the default dict + add. Add `extra_aliases=` that merges, or document the copy-pattern explicitly.	v0.1.1
X4	The rc1 → v0.1.0 migration guide had four documented gaps — different recipes needed Django-specific snippets, ordering was load-bearing in one place, the projection shape was wrong for a required-`code` envelope, and a `request_id=None` patch needed an explicit "top up from your own ContextVar" note.	v0.1.0 doc patch

Two of those (X1, X2) are surface-quality nits — the kit works, but the ergonomics chafed. Two (X3, X4) are real correctness traps — silent override of the default alias table, and migration recipes that worked for FastAPI but not for DRF. Both reports independently flagged all four. That kind of convergence is exactly what dogfooding is for: a single user can rationalize around a sharp edge; two users hitting the same edge means the edge is the bug.

The single-repo findings were sharper.

FastAPI-specific: from_exception(envelope_cls=...) projects the kit's exception details into [{field, message}] per error-list-item. FastAPI's ErrorDetail schema also requires code — so the projection raised a ValidationError at handler time. Cost to diagnose: ~10 minutes. Workaround: drop envelope_cls from the call, translate manually in the handler (six lines). Tracked as a v0.1.1 patch (add code=exc.error_code to the projected items, or add a per-entry callback hook).

Django-specific: the migration guide's verify_envelope_contract example called handler(...).body — that's the FastAPI shape. DRF stores the body at .data, not .body. Doc-only fix; it landed in the same PR as the report.

Smoke proof: the Django report didn't just say the bridge worked. It quoted the response:

GET /api/accounts/me/  (anonymous, 401)

X-Request-Id: 1e43827d5faa4fcfa3551acfcf9caa27
{"request_id":"1e43827d5faa4fcfa3551acfcf9caa27", …}

The same UUID in the response header and in the JSON envelope's request_id field. rc1 returned "request_id":null. v0.1.0 returned the bound value. The bridge — bind_to(request_id_ctx) in a thin BindRequestIdMiddleware slotted immediately after the kit's RequestIdMiddleware — flowed kit → bridge → DRF handler → response. End to end. That's the kind of proof migration reports should ship.

Convergent v0.2 wishlist. Both reports independently asked for the same five things: DjangoSettingsSource (read settings.RESILIENCE instead of forcing env vars), resilience_kit.utils.* (five small modules every boilerplate re-implements: log_sanitization, network, timing, function_logger, data), a GlobalThrottle (process-wide cap; nginx covers it in prod but laptop dev wants the in-process belt), a free-function metrics.record_* shim over MetricsSink to keep adopters' bounded-cardinality guards in place, and multi-alias Redis topology so cache, throttle, and breaker can point at separate instances. That convergence is the v0.2 design spec, half-written for me.

Bugs the Kit Found in Itself

The dogfooding gate is the systemic story. The bugs are the craft story.

`f6cebdd` — outer breaker, inner retry

The bug behind axiom 3. A caller passed retry_on=(Exception,) to the retry decorator. The decorator caught ServiceUnavailableError — exactly the wrong exception. The fix is the _filter_retry_on snippet above. The rule it earned is in axiom 3.

The inverted stack is the one a fresh reader writes first — composition order looks like a style preference until the breaker is OPEN. The _filter_retry_on invariant makes the wrong order fail safely instead of catastrophically: the retry decorator refuses to swallow the breaker's "stop" signal, even when the caller explicitly tells it to.

`bc499da` — `asyncio.Event` rebinding on every restart

RecoveryMonitor held an asyncio.Event created in __init__ and reused across every start()/stop() cycle. asyncio.Event doesn't bind to a loop at construction — it binds lazily, on the first .wait() or .set(). So the first start captured the loop. The second start, on a fresh loop, raised:

RuntimeError: <asyncio.locks.Event object at …> is bound to a different event loop

It surfaced in pytest-asyncio first — two FastAPI integration tests in one session, per-function loop scope. The second test's lifespan opened a fresh loop and called monitor.start(); the Event still held the first test's closed loop. Same shape of bug as Django's runserver autoreload, Gunicorn graceful restart, any ASGI lifespan that opens twice. The fix:

# In src/resilience_kit/recovery.py, RecoveryMonitor.start()
def start(self):
    with self._lock:
        if self._task is not None and not self._task.done():
            return
        # Reassign rather than .clear() so the Event binds to the
        # *current* event loop. asyncio.Event lazily binds to
        # whichever loop first calls .wait()/.set() on it; reusing
        # the prior Event across loops (test harness, restarted
        # server) raises ``RuntimeError: bound to a different event
        # loop``. A fresh Event has no binding yet.
        self._stopping = asyncio.Event()
        self._task = asyncio.create_task(self._run(), name="resilience_kit.recovery_monitor")
    _logger.info("RecoveryMonitor started.")

Six lines changed. The lesson generalizes: asyncio.Event, asyncio.Queue, asyncio.Lock and several siblings capture their loop binding on first use, and .clear() doesn't unbind. Loop-bound state is loop-scoped, not loop-local. Construct fresh on every lifecycle restart; don't cache.

`9f3d846` — `pybreaker` wrapper swallowed the tripping-call exception

The kit wraps pybreaker.CircuitBreaker as one of three breaker backends. On the call that trips the breaker (the one that crosses the failure threshold), pybreaker raises its own CircuitBreakerError to signal "I just opened" — discarding the underlying exception that caused the trip. The caller saw "breaker open" without knowing why it opened. Forensics suffered.

The fix: the wrapper captures the original exception in a small state object before letting pybreaker raise, then re-raises the original on the tripping call. The breaker still opens; the caller still sees the real failure cause. Don't let third-party wrappers eat your forensic signal.

Honourable mentions

aaa5f41 — ASGI header bytes vs str. The request_id middleware did .lower() on b"X-Request-Id". Worked on Starlette's str-typed headers, broke on raw ASGI bytes. Boundary type discipline.
1c8f834 — settings cache leaking the test source between tests, masking what legacy_env_alias actually resolved. The reset helper now restores the default EnvSettingsSource, not the last test's override.
e588e8c — CodeQL flagged a dict whose keys were unioned str | bytes. The fix was three characters. The lesson: take CodeQL findings seriously even when "it works"; the static analyser sees the type-confusion path the runtime tests don't.

These are real fixes from the kit's git history. None of them were dramatic. All of them earned a small piece of the design.

Building the Dangerous Parts First

Two areas in the kit are trivial to get wrong and catastrophic when they fail.

SSRF guard + DNS rebinding TOCTOU

An SSRF guard that just validates a resolved IP is vulnerable to a Time-of-Check-to-Time-of-Use (TOCTOU) race. The attacker's DNS returns a safe public IP at check time; the SSRF guard approves; the underlying HTTP transport does its own DNS lookup; the attacker's DNS returns 127.0.0.1 this time; the request lands on localhost.

The fix is to make the check and the connect share state. AsyncAPIClient resolves the host, validates the IP against the SSRF allow-list, and pins the validated IP into a contextvars.ContextVar. A custom httpx transport reads the pinned IP and connects directly — skipping the second DNS lookup entirely while preserving the original Host header and TLS SNI hostname so cert verification still passes.

The check and the connect now share a single piece of state. TOCTOU bypassed.

Fire-and-forget audit dispatch

Audit logs can't bring down the application. The pre-kit boilerplates wrote audit rows synchronously — if the database hiccupped, the request failed. The kit replaces that with a decoupled, fire-and-forget pipeline:

An in-memory, bounded asyncio.Queue (default 10,000 events) acts as a buffer.
A background worker flushes batches to the configured audit backend (postgres, stdlib_logging, or a custom entry-point backend).
If the database is down, the dispatcher degrades gracefully. It retries with backoff; if it ultimately fails, it logs the batch to stderr and increments a dropped metric. The app survives.

The audit dispatcher is also one of the loop-owned resources from axiom 4 — the Django adapter spawns it inside its daemon thread's private loop and drains it via atexit on the same loop before closing.

Verification: Testing What Unit Tests Can't See

The test suite splits three ways. Unit tests verify logic in isolation. Contract tests verify the same logic works against every backend. Integration tests verify the whole stack survives chaos.

Contract tests: the same assertion, every backend. The kit defines multiple backends — memory, Redis, pybreaker for circuit breakers. The same test suite runs against all of them, parametrized by pytest.mark.parametrize("backend", …). A Lua script in redis_impl.py that diverges from the in-memory contract fails the same assertion that passes against memory_impl.py. The contract suite catches it the moment it ships.

SSRF TOCTOU validation. The integration test (tests/integration/test_dns_rebinding.py) mocks socket.getaddrinfo with a _RebindingResolver that returns a safe public IP on the first call and a private IP on the second. The SSRF guard validates the first IP and pins it. The connection should use only that pinned IP — no second resolution. The test asserts the underlying transport's request URL carries the pinned IP, not the attacker's late-arriving private one.

Recovery monitor validation. The integration test (tests/integration/test_recovery_monitor.py) spins up a real Redis with testcontainers, calls docker_client.api.pause(container_id) to freeze TCP without killing the connection, verifies that throttles degrade to in-memory and flag degradation in metrics, then unpauses and verifies the monitor restores the Redis-backed providers within the 5-second exit-gate window. This test is what caught the bc499da bug — the second invocation of the monitor across pytest's per-function loops blew up exactly as described above.

The kit ships with 52 test modules across unit, contract, and integration suites.

What I'd Do Differently

Four regrets from the critical path:

Chaos and failure injection should come before v0.2. I built the core under happy-path assumptions, then spent the adapter phase hardening it. Testcontainers, container-pause cycles, Redis-down scenarios — these should have been mandatory from day one, not post-release validation. Every failure mode I caught during adapter work would have reshaped a primitive's design if I'd seen it earlier.
CI matrix (Python 3.11, 3.12, 3.13) should be commit-one, not late hardening. I assumed 3.11 compatibility, then discovered version-specific differences in asyncio behaviour and stdlib deprecations during release prep — exactly the work a GitHub matrix would have done on every PR for free. Fixes were mostly mechanical; the cost was the discovery cycle.
Contract test suite should precede any backend implementation. I wrote backends (memory, Redis, pybreaker) and then wrote tests to validate them. Tests written after backends encode the backends' assumptions as the spec, not the spec's expectations as the test. Contract tests written first force you to commit to the behaviour before you commit to the implementation — and they catch backend-specific divergence at the boundary rather than during integration. I got there eventually; getting there first would have been cheaper.
mypy --strict should be enforced from commit-one, not applied retroactively. I shipped the first half of the project without strict type hints, then made it a release gate. Retrofitting types to async code with complex control flow is error-prone, and the rewrite surfaced structural mismatches between Protocol declarations and the backends meant to satisfy them. Strict types from the start force the design right the first time.

These are design regrets, not dogfooding bugs. The dogfooding bugs fed v0.1.1 and v0.2. The design regrets are about how I'd budget time on the next from-scratch infrastructure project.

What's Next

The convergent v0.2 wishlist from both migration reports is half-written for me:

DjangoSettingsSource — read settings.RESILIENCE directly instead of forcing env-only config. Both reports rank this P1.
resilience_kit.utils.* — five small modules (log_sanitization, network, timing, function_logger, data) every boilerplate re-implements. Promoting them into the kit saves ~900 LOC per consumer.
MetricsSink cardinality contract — bound the labels at record_* time, not after a Prometheus cardinality explosion in prod.
GlobalThrottle (Valkey-Lua) — process-wide cap. Nginx covers it for fleet deployments; the in-process belt is for laptop dev and single-pod deploys.
Multi-alias Redis topology — RESILIENCE_REDIS_URLS__<alias> so cache, throttle, and breaker can point at separate instances.

The v0.2 exit gate is the same rule, raised: both boilerplates re-test against a v0.2 pre-release and score ≥ 8.5 / 10 — half a point higher than 0.1.0.

Full roadmap: ROADMAP.md.

I started by trying to delete duplication. I ended up with five axioms, a dogfooding release rule, and a set of bug-earned invariants that are reusable far beyond this kit. The duplication was the symptom. The axioms are the fix.

Notes & references

Each axiom and dangerous-parts decision has a one-page ADR in the repo. If you want the long-form reasoning — alternatives considered, consequences, usage notes — these are the load-bearing ones:

Topic in this post	ADR
Protocols, not ABCs	`0001-protocol-not-abc`
Hand-rolled retry over `tenacity`	`0002-handrolled-retry-not-tenacity`
One package, many extras	`0003-single-package-with-extras`
Entry-point backends + precedence chain	`0004-entry-points-for-third-party-backends`, `0009-entry-point-precedence-chain`
Fire-and-forget audit dispatch	`0005-fire-and-forget-audit`
Outer breaker, inner retry	`0006-outer-breaker-inner-retry`
DNS pin via `ContextVar` (TOCTOU)	`0007-dns-pin-via-contextvar`
Fernet env-guarded keys	`0008-fernet-env-guard`
FastAPI adapter shape	`0010-fastapi-adapter-shape`
Django sync/async bridge	`0011-django-sync-async-bridge`

Migration reports: docs/m8b-upgrade-reports/ — SUMMARY.md, fastapi_boilerplate.md, django_boilerplate.md.

The adapter is not the kit. The kit is what survives without the adapter.

Source Code: resilience-kit GitHub Repository
Documentation: Design Doc (PRD) | Architecture (LLD) | ADR Index | Migration Reports

Building toymq: a from-scratch persistent message broker in Go

Prajwal Mahajan — Tue, 09 Jun 2026 20:04:45 +0000

A retrospective on toymq — a single-node persistent message broker in Go, recreated by hand to understand one of the smallest functional units in a distributed system.

TL;DR — and the bug you should remember

Build the dangerous parts first. Build the thing that, if it's wrong, forces a rewrite of everything above it. Then build the thing above it.

The worst bug of the project took six commits to surface and an integration test to catch: a uint64 consumer offset where 0 meant both "message id 0 was acked" and "never acked." Unit tests never restarted the broker between Ack and Subscribe, so the collision stayed invisible. Zero values are not sentinels. That's the lesson the post earns.

The numbers: ~10k lines of Go, 17 ADRs, four binaries, 90.3% test coverage, a chaos harness that pushed 14,000 messages through three SIGKILL/restart cycles with zero acked-message loss. v1.0 in four days; v1.3 plus post-release hardening in another two.

The order:

WAL → wire protocol → broker → session → server → cmd wiring
   → integration tests → chaos → post-v1.0 hardening

The point of this post is the order, not the code.

What toymq is, and isn't

A learning artifact. Single node. No replication. No auth. No TLS. Throughput tops out at a few hundred messages per second because every publish does a per-message fsync. If you need a real broker, reach for NATS or RabbitMQ. If you want to understand what a broker is — what durable actually means, what at-least-once costs, what crash recovery requires — build one. That's what this post is about.

Why "build the bottom first" is non-obvious

Every tutorial does the opposite. You build a server, accept connections, parse a request, stub a handler, add real handlers, add persistence last. By the time you reach durability, you've shipped a wire protocol that assumes you don't have one, a handler API that assumes nothing crashes, and a test suite that mocks the part of the system that matters.

toymq inverts that. The first commit isn't a server — it's a framed record format and a CRC check. Day 1 is "what is on disk after a crash" and nothing else.

This works for a selfish reason: the bottom of the stack, if I get it wrong, forces a rewrite of everything above it. A storage bug means the broker is wrong. A protocol bug means every client and the broker are wrong. Risk shrinks as you go up. So you spend the budget for being wrong at the bottom, while the budget is highest.

Durability: what should "on disk" actually mean?

The storage piece of any persistent system is the write-ahead log — a file you append records to, fsync, and never go back to mutate. On crash, you scan it from the start and rebuild your state.

The one rule for the WAL: the on-disk format is a contract. Anything that lands on disk has to survive forever, because the recovery scan will assume it. Changing the format later means writing a migration.

On-disk layout

data/
└── topics/
    ├── orders/
    │   ├── segment.log     ← append-only WAL, one record per PUB
    │   └── offsets.json    ← per-consumer state, written by debouncer
    └── events/
        ├── segment.log
        └── offsets.json

One directory per topic. Two files per topic. No metadata file, no index, no manifest. The recovery scan walks every segment from offset zero on Open. Scans are O(disk) but correct by construction — the WAL is its own truth, and a manifest would be a second consistency problem on top of the first.

The record frame

Field	Type	Size	Notes
`length`	`u32`	4 bytes	Bytes that follow, excluding self
`msg_id`	`u64`	8 bytes	Monotonic per topic
`ts_ns`	`u64`	8 bytes	Append timestamp, ns since epoch
`key_len`	`u16`	2 bytes	0 if the message has no key
`key`	bytes	`key_len` bytes	Dedupe key
`payload_len`	`u32`	4 bytes	Payload length in bytes
`payload`	bytes	`payload_len`	Message body
`CRC32`	`u32`	4 bytes	Checksum over all preceding bytes

Deliberately boring. The interesting part is what's not there: no version byte. Adding a version byte later is a one-line migration; defending the wrong version byte forever is not. Don't add knobs the format doesn't yet need.

`fsync` is the durability commit point

Log.Append is the only function in toymq that calls fsync. Every PUB goes through it. Every OK the broker emits is, by construction, preceded by a successful fsync of the corresponding record:

The committed-offset update happens after fsync. Any reader that sees committedOffset == X is guaranteed the bytes through X are on stable storage. Cost: ~1–2 ms p99 on commodity NVMe. The correctness budget gets spent here, not on throughput tricks I can't defend. A broker that loses an acked message is not a broker.

How this compares to Kafka

Kafka does almost none of this. Its durability story is page cache + replication: trust the OS to flush eventually, trust the replicas to catch up. It pushes hundreds of thousands of messages per second per broker as a result. toymq has neither replicas nor the throughput budget to assume the page cache wins, so it trusts fsync directly. Different problem, different answer — and the gap between those answers is exactly what hand-rolling teaches you.

The first nasty bug landed here, too: a make([]byte, payloadLen) that ran before checking payloadLen <= maxPayload. A malicious client could OOM the broker with a single packet claiming a 100 GB payload. The fix is one line; the rule is general — validate before allocating, always.

The wire protocol: agreeing on what happened

Once the storage layer holds, the next question is how clients and broker talk. A protocol is the contract that lets the two ends disagree about when something happened (network is unreliable) without disagreeing about whether it happened.

toymq's protocol fits in a paragraph: a command word, a few arguments, a length-prefixed payload. The full PUB happy path, end to end:

The OK only crosses the network after fsync returns. That's the durability promise made visible: the client never sees an OK for a message that isn't on disk.

Two design choices shape everything above this layer:

A sealed Command type. The parser dispatches on an interface with an unexported marker method. You cannot add a new command type without editing the file the parser lives in. The compiler keeps the protocol and its handler in sync — if you add a command and forget to handle it, the build breaks. Free safety.

Clean EOF vs torn header. The parser distinguishes a stream closed cleanly between commands (propagate io.EOF) from a stream closed mid-line (return ErrBadFraming). That distinction recurs everywhere downstream: the session loop uses it to tell "client gracefully disconnected" from "client crashed and dropped." Most tutorials skip this and pay for it forever.

Brokering, in two parts: structure vs semantics

A broker has to do two things: route messages from producers to consumers, and guarantee something about the delivery (at-least-once, in toymq's case). I split this into two branches and the split mattered.

Structure came first: topics (auto-created on first publish), a WAL per topic, an LRU dedupe index keyed by (producer-id, msg-id), in-memory consumer state. Acceptance bar: "you can publish a message and consume it; if the broker restarts, recovery works." No visibility timeouts, no NACKs.

Semantics came second. Every message a consumer has been told about lives in one of three states:

A message can bounce between Pending and Inflight many times before reaching Acked — that's the at-least-once contract. The chaos suite's no-loss invariant verifies that every acked MsgID reaches at least one consumer's seen set. Duplicates above one are allowed; loss is not.

The redelivery ticker is the load-bearing piece, and it forced the rule that became the hot-path discipline for the entire broker:

Never send while holding the inflight lock.

c.inflightMu.Lock()
pending := append([]*Message(nil), c.inflight...)
c.inflightMu.Unlock()

for _, msg := range pending {
    select {
    case c.SendCh <- msg:        // safe — no lock held
    case <-ctx.Done():
        return
    }
}

A blocking send on a channel, while holding a lock the redelivery scan also wants, is deadlock bait. Take the snapshot under the lock, release it, then send. The symmetric rule on the ticker side: the scan does hold the lock for the full pass — releasing per-entry lets a concurrent ACK delete an entry mid-iteration and panic the map. -race finds the second one in five seconds; code review catches the first.

I tried to do structure + semantics in one branch first. I rolled it back at the second self-merge conflict. The two-merge rule: if a branch conflicts against itself twice before it's done, the branch is too big. Split it.

Concurrency: serving many connections without corruption

The broker is one process; clients are many. The design question is how many goroutines per connection, and who owns what.

toymq's answer: three goroutines per session, one channel of truth between them. A fourth — the broker's per-subscription delivery worker — lives on the broker side but writes into this session's outbound channel.

Session R owns the socket's bufio.Reader. Nothing else reads.
Session W owns the socket's bufio.Writer. Nothing else writes.
Session H dispatches parsed commands to the broker and feeds responses into outbound.
runDelivery is the broker's delivery goroutine for this consumer; it also writes into the session's outbound channel.

outbound is the only place where two goroutines (H and runDelivery) might race to write the socket — and the channel itself serializes them, no lock needed. bufio.Writer has no concurrent-safe contract, and you don't lock around it; you funnel through a channel. Closing the socket, draining outbound, and joining all goroutines becomes one operation: cancel a context, wait on a WaitGroup.

The rule that came out of this is sharp and reusable: every go-spawned goroutine must be accounted for in exactly one WaitGroup or doneCh. A "WaitGroup race" in the listener — wg.Wait returning before a goroutine that called wg.Add had a chance to register — was the bug that taught me to write the rule down explicitly.

The TCP accept loop adds two subtleties most servers get wrong: an EMFILE-aware exponential backoff (a burst of connections can exhaust file descriptors and tight-loop the broker), and an explicit listener.Close() on context cancel (without it, Accept() blocks forever and shutdown hangs).

cmd/toymq keeps main.go thin: a separate internal/config package owns flag validation, a testable run(ctx, args, stdout, stderr) int lets tests drive shutdown via context cancellation, and signal.NotifyContext wires SIGTERM/SIGINT into the context in one line. The smoke test boots run, sends a fake SIGTERM, and asserts a clean exit. It catches "shutdown leaks a goroutine" the moment it happens.

Testing what unit tests can't see

By this point every package had unit tests. Coverage was 78%. -race clean. But the most interesting bugs in any networked system live in the seams — what happens when the broker sends a 300-byte MSG, the client receives bytes 1..200, and then disconnects?

I built in-process integration tests next: a real broker + server on a kernel-assigned TCP port, a stripped-down test client that queues incoming MSGs so asynchronous deliveries don't deadlock the response-reading. Six scenarios: round-trip ACK, 1000-message restart, visibility-timeout redelivery, NACK redelivery, dedupe, subscribe takeover. Coverage went from 78% to 90.3%.

The suite caught the worst bug of the project on its first run. The test:

Publish message id=0. Subscribe. Receive MSG. ACK.
Restart the broker.
Subscribe again. Expect no replay.

It failed. Message id=0 replayed. With id=1 as the first message, no replay. The bug was specifically the zero msg-id.

Root cause: consumer state on disk stored a single lastAcked uint64. Recovery treated lastAcked == 0 as "never acked" (the Go zero value) instead of "acked msg id 0." On the first restart after acking the very first message, the broker couldn't tell the difference. The fix adds a hasAcked bool and persists both:

type ConsumerState struct {
    LastAcked uint64
    HasAcked  bool   // ← the field that should have been there from day 1
}

The Go-language lesson: zero values are not sentinels. They collide with valid data the moment your domain is non-empty. The systems lesson is sharper — the bug existed for six commits before anyone noticed. Unit tests never restarted the broker between Ack and Subscribe. By the time integration tests ran, the broken state was sealed in code-review history weeks earlier. The integration test caught it on its first run.

Then I added chaos: a supervisor that SIGKILLs the broker on schedule and restarts it; a producer that keeps publishing through restarts; a consumer that records every msg-id it ever sees. Invariant: for every msg-id the producer got an OK for, the consumer must receive it at least once, eventually. A 90-second smoke pushes 14,000 messages through three SIGKILL cycles with zero acked loss. That's not a proof of correctness — it's evidence the WAL + fsync + offset design holds up under realistic crash patterns.

Chaos also found a data race in the chaos test itself — a bytes.Buffer capturing stderr being written by the supervisor and read by the assertion. The broker was correct; the test had the race. -race is for everything you ship, including tests. A race in test infrastructure can mask real bugs or manufacture phantom ones.

After v1.0: the part where the work gets easier

Once the bottom half held, the top half was craft, not danger. A Go client library (pkg/client) on top of the protocol, with a single read goroutine that demuxes incoming frames by type — separate channels per call would have raced two reads on the socket, a deadlock recipe. A CLI (toymqctl) and a latency harness (toymq-bench). A Bubble Tea TUI (toymq-tui) with one sharp lesson worth keeping: boolean state for "is X active?" is a smell. A bool for "is a modal open?" broke the day the second modal shipped on top of the first. A modal stack fixed it. Lists, stacks, enums — anything but a boolean — almost always model the actual shape.

Then Prometheus metrics and OpenTelemetry tracing, with the design rule: observability does not change behavior. No locks, no goroutines, no allocations on the hot path. prometheus.CounterVec.WithLabelValues is cached; otel.Tracer().Start is a no-op when no provider is attached.

Finally, CI hardening after main went red twice in a row with gofmt drift. Two recurrences is the line where you stop fixing instances and start fixing the system: a Makefile, an opt-in pre-commit hook, golangci-lint with a conservative ruleset, and a Go 1.25 + 1.26 CI matrix. The linter's first run flagged 31 things; seven were real bugs (commited, exhuastion, three dead helpers, one cap-shadowing parameter). A linter that flags 30 things on its first run is doing its job.

ADRs as crystallization, not ceremony

Seventeen ADRs sounds like ceremony. It wasn't.

The rule: write the ADR the moment the decision is forced by code, never before. Not when I had a hunch. When I had to type "we are doing it this way" into a function, then the ADR went next to it.

ADRs written before code force the code to fit the ADR, which means defending decisions made when you knew the least. ADRs written at crystallization record what the code already decided. If the code later changes its mind, you supersede the old ADR — you don't try to retroactively justify it. The point is that the ADR is always honest about when the decision was made.

What I'd do differently

Run chaos one branch earlier. Integration tests didn't find what chaos found one branch later. Chaos was the right tool, just deployed one branch too late.
CI matrix from day one. A Go 1.25 + 1.26 matrix takes 30 minutes to set up and prevents an entire class of "works on my machine" failure. It should have been part of the first chore(ci) commit, not a retrofit at the end.
Wire trace context through the protocol from v1.3. Spans today are root spans inside the broker. Cross-process correlation needs a TRACEPARENT line in the wire format. Wire-format changes are exactly what you regret deferring.
Treat zero values as a code smell from day one. The lastAcked bug was the most embarrassing of the project. Whenever the answer is "use 0 / "" / nil / -1 to mean absent," the right answer is (value, present bool).

What's next

Persisting the dedupe LRU to disk without losing the boring-by-design property of the WAL. Right now the LRU is in memory only — survives one broker lifetime, lost on SIGKILL. The fix mirrors the atomic-swap pattern used for offsets; the interesting question is what happens on a torn write of the LRU snapshot itself, since unlike the WAL there's no CRC-framed record stream to scan. That's the next post.

After that, the series moves to tinykv (a Redis-subset KV store in Go) and tinyraft (a 3-node Raft consensus cluster). Same risk-first sequencing. Same ADRs-as-crystallization. The arc ends where you compose them: replicated state machine semantics on top of a real consensus log.

The one-line version

If a tutorial tells you to build the server first, you've already started in the wrong place. The storage is the project.

Source: prajwalmahajan101/toymq. Deeper docs (with every diagram in this post and more): ARCHITECTURE.md, PERSISTENCE.md, CONCURRENCY.md, REDELIVERY.md, FLOWS.md. ADR index: docs/adr/. Open ideas: IDEA.md. Corrections welcome — open a discussion or file an issue.

DEV Community: Prajwal Mahajan

Building toykv v2: seven features that didn't earn the major, one refusal that did

TL;DR — the decision, and the check, you should remember

Why the order is the design

Seven rules v2 earned

The dangerous parts of v2, one at a time

Lists and hashes — the header byte you rewrite under the old binary's feet

Honourable mention — the "flaky" test was telling the truth

AUTH + TLS — the feature that shipped in one boolean

SCAN — you can't port Redis's cursor to a Go map

Protected mode — the refusal that earns the major

Observability, then release — no-op providers are not free, and the auth review missed a pre-auth DoS

After the tag — writing the manual to test it found two more

Honourable mentions — three smaller bugs that earned a line

Proof it composes

Decision records as crystallization — the supersede trail

What I'd do differently

What's next

The one-line version

Building toymq v2: eight features, one durability contract, and the bug that proved it held

TL;DR — the decision, and the bug, you should remember

Why the order is the design

Five rules v2 earned

1. The durability contract is sacred. A feature negotiates around it, never through it.

2. Every feature is designed around the seam the distributed future needs.

3. Breaking an old client must fail safe or opt in — never silently misbehave.

4. Backpressure is about the slow half you aren't looking at.

5. Observability that needs its own dependency in the hot path is the wrong observability.

The dangerous parts of v2, one milestone at a time

M3 — the handshake response that got dropped 40% of the time

M4 — Topic becomes a router, and the flat layout survives

M5 — a wake channel, because sync.Cond can't say no

M6 — three features that had to not corrupt each other

M7 — the continuation that the parser refused

Proof it composes

ADRs as crystallization — the supersede trail

What I'd do differently

What's next

The one-line version

Building toykv: a from-scratch persistent KV in Go, and why I took the opposite call from toymq three times

TL;DR — and the rule I want you to remember

What toykv is, and isn't

The crash matrix

Inversion 1 — a version byte you'll thank yourself for, and absolute deadlines

Absolute deadlines, never relative durations

The AOF on disk

Inversion 2 — write the bytes you'd send

Inversion 3 — RESP because the harness is free

Honourable mention 1 — the dual-write bufio bug

Honourable mention 2 — the test was wrong, not the code

ADRs as crystallization — and the ADR I almost didn't write

What I'd do differently

What's next

The one-line version

Building resilience-kit: A Python Resilience Kernel Forged in Production

TL;DR

Five Axioms

1. One package, many extras. Not a forest of micro-packages.

2. Protocols, not ABCs.

3. Outer breaker, inner retry. Composition order is correctness.

4. Async-first with explicit sync bridges. Never trust auto-bridging.

5. Adapters are dumb wiring. If your adapter has business logic, your primitive is wrong.

The Dogfooding Gate

What the Migration Reports Found

Bugs the Kit Found in Itself

f6cebdd — outer breaker, inner retry

bc499da — asyncio.Event rebinding on every restart

9f3d846 — pybreaker wrapper swallowed the tripping-call exception

Honourable mentions

Building the Dangerous Parts First

SSRF guard + DNS rebinding TOCTOU

Fire-and-forget audit dispatch

Verification: Testing What Unit Tests Can't See

What I'd Do Differently

What's Next

Notes & references

Building toymq: a from-scratch persistent message broker in Go

TL;DR — and the bug you should remember

What toymq is, and isn't

Why "build the bottom first" is non-obvious

M5 — a wake channel, because `sync.Cond` can't say no

Honourable mention 1 — the dual-write `bufio` bug

`f6cebdd` — outer breaker, inner retry

`bc499da` — `asyncio.Event` rebinding on every restart

`9f3d846` — `pybreaker` wrapper swallowed the tripping-call exception

`fsync` is the durability commit point