DEV Community: Hugo Vantighem

When "it works in prod" becomes worse than "it works on my laptop"

Hugo Vantighem — Mon, 01 Jun 2026 09:28:16 +0000

Every engineering team accumulates technical debt. That's normal.

The dangerous moment isn't when the debt appears. It's when everyone agrees it's there — and decides to keep building on top of it anyway.

I've seen that shift happen in almost every engineering team I've worked with. It doesn't announce itself. It arrives quietly, dressed as pragmatism. And by the time you notice it, it's already become the default way the team operates.

It starts with a gap in knowledge. And that's completely fine.

The first error is innocent

Making design or architecture mistakes isn't a problem. We all do the best we can with the knowledge we have at the time. That's not a failure — it's how engineers grow. You learn by building, and sometimes you build something wrong before you know what right looks like.

The first bad design decision is usually just that: a knowledge gap. Nobody in the room knew better. They shipped with what they had.

Then someone learns something

Time passes. The team gains experience. Someone reads the right book, works through a gnarly bug, or joins from a company that had solved this exact problem already. The gap closes.

And with it comes the moment that defines a team's culture.

That person — maybe you — points to the design flaw. Not to criticize, not to reopen old wounds, but because they can see it clearly now and want the team to move forward. They bring the argument. They show why it matters.

And here's where things split.

Two possible answers

The first answer is a technical discussion. Maybe the team agrees immediately. Maybe they push back, challenge the framing, propose a different fix. Maybe they decide the cost of changing is too high right now and park it for later. Any of that is fine. The conversation is alive. The team is reasoning together about its own code.

The second answer is: "Yeah but it already works like this in prod. Why change it?"

The conversation stops. The technical debate is over before it started. And something quietly shifts.

The real tipping point

When that second answer happens, the error hasn't disappeared. It's now fully visible — known, understood, and acknowledged by everyone in the room. But instead of becoming a problem to solve, it becomes a fact to live with.

Known.
Understood.
Accepted.
Normalized.

That's the tipping point. Not the original mistake — that was just a knowledge gap. The tipping point is the moment the team decides, consciously, that a known flaw is the way things are.

We've moved from "we didn't know better" to "we know, and we're choosing not to act." Those are fundamentally different things. The first is human. The second is a choice about what kind of team you want to be.

The shield protects everything — including what it shouldn't

The pattern is corrosive because "it works in prod" gets used across the entire spectrum — and the severity of what it's shielding rarely changes the answer.

Sometimes it's a structural decision that's now load-bearing: real cost, real risk, the resistance is at least understandable. The mistake is using the phrase to close the conversation rather than as a factor in a real discussion about when and how to address it.

But then there's the case that should never be shielded.

Design flaws with real business risk attached. A service that writes to two stores. Someone points out the write isn't atomic — a crash between the two leaves them permanently out of sync. The answer comes back: "We've been running it for two years." Not because the concern is wrong. Because it hasn't exploded yet.

A data model that creates silent desync under concurrent load. A sequence that can leave the database in a half-committed state on crash. An architecture that, under the right conditions — a spike, a timeout, a race — loses data.

It works in prod. For now. Because the conditions that trigger it haven't aligned yet — or because they have, quietly, and nobody noticed.

At this level, "it works in prod" isn't pragmatism. It's a decision to accept an unquantified risk indefinitely, on behalf of users who don't know they're carrying it.

The system isn't stable.

It's stable so far.

The team knows the difference.

How it compounds

What makes this dangerous isn't any single decision. It's what comes after.

Because the next feature gets built on top of the flaw. Then the next one. Each decision taken in isolation is defensible — you're adapting to what exists. But you're also adding weight to a foundation that everyone now knows is wrong.

And when the cost of fixing finally becomes impossible to ignore, the same logic applies again: "It would cost too much to redo now."

A new trade-off. Based on the previous one. Which was itself based on the one before.

This is how absurd architectures are born — not from one catastrophic decision, but from a temporary shortcut that got normalized, then depended upon, then treated as untouchable. Accidental complexity doesn't arrive all at once. It compounds, layer by layer, until it becomes the team's technical culture.

Why "it works in prod" is more dangerous than "it works on my laptop"

Every developer knows what "it works on my laptop" means: the work isn't finished. Nobody treats it as a conclusion — everyone understands it as the beginning of the real work.

"It works in production" carries the opposite social weight. It sounds like evidence. It carries the authority of real users, real load, real money. And it gets used for exactly the same purpose — to stop the conversation.

The difference is that one is obviously incomplete. The other gets mistaken for proof.

That's what makes it more dangerous. Something that breaks locally is obviously broken. Something that works in production has inertia on its side — real users depend on it, changing it carries risk, the system resists being touched. That inertia is real. But the moment it becomes a reason to close a technical conversation rather than inform one, the phrase has shifted from a statement about stability to a shield against growth.

What you lose — and who you lose

The real cost isn't code quality. That's the surface.

What you lose is the team's ability to reason honestly about its own system. The questions that are supposed to stay open get closed by precedent. And once the culture of closing them is established, it applies to the next problem too. And the one after.

When "it works in prod" becomes the team's default posture, you can map it exactly onto what "it works on my laptop" reveals about an individual developer: someone who has stopped asking whether it's right, and started asking only whether it's currently not broken. At the individual level, that's a conversation. At the team level, it's a culture.

The tell is simple: watch what happens when someone raises a known problem. Does the team ask "how do we address this?" or "why are we even talking about this?" The first compounds knowledge. The second compounds debt — and has decided that's fine.

The engineers who care most are also the ones most likely to leave when the answer is always a wall. Not because of the debt itself. Because of what the wall says about the team.

The only thing that's not negotiable

Every team carries debt they can't pay down immediately. Every team makes trade-offs under pressure. That's not the failure.

The failure is letting the conversation die.

The technique is learnable. You can hire your way out of a knowledge gap, train your way out of a skill gap, refactor your way out of a design gap. None of those require the team to be perfect — just willing.

The willingness to keep questioning, to hear "we built this wrong" as useful information rather than a threat — that's not a technical skill. It can't be reviewed, tested, or measured. But it's what determines whether a team compounds its knowledge or just compounds its debt.

Every team accumulates debt. The difference is whether it remains a conscious trade-off or becomes an unquestionable fact.

The day a team stops discussing known flaws isn't the day the architecture fails.

It's the day learning stops.

Read-Modify-Write isolation in NoSQL: the distributed-lock hell.

Hugo Vantighem — Thu, 28 May 2026 06:36:00 +0000

In part 1, the single-document case was easy. In part 2, two documents brought Write Skew, and we saw that even a native ACID transaction — snapshot isolation — lets it through.

So teams reach for the reflex fix: a distributed lock — Redis-based, often a Redlock-style implementation. Acquire a lock on a key, do your Read → Modify → Write, release. On paper, you've finally serialized the critical section — operationally, at least. In practice, you've stepped on three mines.

1. Network latency

Every guarded transaction now makes extra round-trips to Redis — before and after hitting your NoSQL store.

You've doubled your coordination surface and taken a hard dependency on a second system being up, reachable, and fast on the hot path of every write.

The "fast" database is now gated by the lock service. And the coupling bites harder than the average latency suggests: every Redis tail-latency spike becomes your write-latency spike — your p99 inherits Redis's p99 — and if Redis fails over mid-transaction, the lock you think you're holding can effectively vanish on the new primary, dropping you straight into the corruption case below.

2. Deadlock

You can dodge deadlock entirely with a single coarse lock — but then every writer serializes on it, and you've thrown away the very concurrency you reached for NoSQL to get.

So to keep throughput you go fine-grained, one lock per resource — and the moment an invariant touches more than one key (across this series, it always does), deadlock is back on the table:

Transaction A locks key X, then needs Y.
Transaction B locks Y, then needs X.
Both block until timeout or intervention.

The textbook cure — real deadlock detection, maintaining a wait-for graph across every lock holder and breaking cycles as they form — is a distributed-systems project in its own right: not something you bolt onto a cache you reached for precisely to save engineering time.

So nobody builds it.

Instead teams impose a standing discipline: always acquire locks in the same canonical order — sort the keys by document ID, lock them in that order, in every code path, every time, forever.

One writer that grabs them out of order — a refactor, a new feature, a teammate who never got the memo — and the deadlock is back. That convention, plus the hopeful timeouts bolted on for when it slips, is the exact accidental complexity you adopted NoSQL to escape.

The cognitive load you tried to delete just came back wearing a Redis sticker.

3. The TTL dilemma

A distributed lock needs a TTL, or a crashed holder blocks everyone forever.

But the moment you set one, you've broken the one guarantee that mattered: you lose any coordination between the lease and the real execution time.

The lock expires on a wall clock; your transaction finishes whenever it finishes — GC pause, slow disk, a network hiccup — and those two clocks have no idea the other exists.

So there is no safe value:

TTL too short → the lock expires mid-transaction. A second writer walks in, and you get exactly the corruption the lock was supposed to prevent — now with a false sense of safety.
TTL too long → a crashed pod blocks the resource until expiry, and your throughput tanks under the very contention you were trying to handle.

No TTL value alone can guarantee correctness across all operating conditions.

No single value is correct across all loads — because correctness constraints and throughput constraints are in direct tension, and the TTL is the single knob you're forced to trade them off against.

This is the heart of the well-known Kleppmann-vs-Redlock debate: a lock that can expire on its own is not a sound mutual-exclusion primitive when you need it for correctness, only for efficiency.

Using it to protect an invariant means betting your data integrity on a timeout.

So are we doomed?

Are we really condemned to choose between the latency and deadlocks of locks, and the silent corruption of going without?

No. There is a fourth path — still within NoSQL, with no distributed lock and no external coordination service — that delivers strict isolation against inter-document Write Skew.

An approach where either everything commits, or everything rolls back, with the invariant intact. It doesn't make the contention disappear — nothing does — it just stops bolting a Redis layer on top to manage it, and collapses the coordination back into the database itself.

That trade is the whole point, and I'll lay it out honestly.

I laid out the full mechanics — the pure-NoSQL pattern almost nobody reaches for — in the next article.

Read-Modify-Write isolation in NoSQL, part 2: When the invariant spans multiple aggregates.

Hugo Vantighem — Wed, 27 May 2026 05:45:00 +0000

In part 1 we saw the single-document case, where optimistic locking saves you with a simple version field. Now we cross the line that breaks that comfort.

Let me make it concrete. Your product sells seats, and an organization buys a license capped at 100 seats. Those seats are spread across many Teams, and each Team is its own aggregate with its own lifecycle. You can't stuff the list of all teams into one document: it grows unbounded and violates every aggregate-design instinct you have.

So your invariant is a sum across aggregates:

The sum of seats over all Team aggregates must never exceed 100.

To enforce that on every "add seats to a team" operation, the honest move is to read the current state across every Team and sum it — a fan-out that scans N Team documents and gets slower as the org grows. Painful, yes. But surely reading the real, current teams inside a transaction is at least correct? That intuition is the trap.

Now the operation is a textbook Read → Modify → Write — but watch what each step touches. You read and sum the Team documents as your guard, confirm there's room, then write the 8 seats onto one Team aggregate (Team Alpha, Team Beta — each its own document). The check reads a set of documents; the mutation lands on one of them. And that's where it gets dangerous.

The anomaly: Write Skew

Two requests run concurrently. The teams currently sum to 90.

guard: Σ seats over all Team docs = 90   (max 100)

  Tx A   reads all teams → Σ = 90 → 90+8 ≤ 100 ✅ → writes Team Alpha (+8) → COMMIT
  Tx B   reads all teams → Σ = 90 → 90+8 ≤ 100 ✅ → writes Team Beta  (+8) → COMMIT
         └─ each reads its OWN snapshot — neither sees the other's in-flight
            write. Two DIFFERENT Team docs → no write-write conflict to abort.

  Σ seats across Team aggregates = 106   >   license cap 100
  Nothing collided, nothing was overwritten.
  The invariant dies between the documents. That's Write Skew.

Each transaction read a valid state and made a locally correct decision — yet the real total is now 106, and you've oversold a 100-seat license. The subtle part the trace makes visible: the damage isn't an overwrite. It lives between the documents, because both transactions validated against a 90 that didn't yet include the other's write.

This is Write Skew, and it's exactly where naive optimistic locking gives up. You'd put a version on the Team document — but Alpha and Beta were never contended; they're different documents. The contended truth is the seat invariant itself.

💡 Lost update vs. Write Skew — the line this whole series turns on.
Part 1's bug was an overwrite: two writes hit the same document, one erased the other, and the stored value itself came out wrong (10 instead of 11). This is the opposite. Nothing is overwritten — the two writes land on different documents, both commit cleanly, and every document stays internally consistent. The invariant breaks in the gap between them. Same Read → Modify → Write race, a strictly nastier failure: there's no torn document for a code review, a unit test, or the database to catch.

A precise word on isolation levels

Wrap the whole thing in a native MongoDB multi-document transaction (≥ 4.0 on replica sets, 4.2+ on sharded clusters) and you get snapshot isolation: every transaction sees a consistent point-in-time snapshot of the database. That genuinely eliminates the classic ANSI read anomalies — dirty reads, non-repeatable reads, and even phantoms (the snapshot is frozen, so rows born after it are simply invisible) — plus single-document lost updates.

What it does not give you, on its own, is full serializability — so it can't stop Write Skew. The subtle part most people miss: your clean snapshot isn't enough, because the concurrent transaction is reading its own snapshot — stale relative to yours — and writing based on it. Two consistent snapshots, two valid decisions, one broken invariant.

Here's the mental key most devs skip: the invariant is never encoded as a write conflict. WiredTiger (MongoDB's storage engine) only aborts a transaction when two of them write the same document. Tx A wrote Team Alpha, Tx B wrote Team Beta — different documents, so there's nothing for WiredTiger to abort. The constraint lives in your head and your validation check, not in the data the engine is watching for conflicts. Snapshot isolation protects the bytes; it can't protect a rule it was never told about.

So the fix — whatever you reach for — has to do the one thing the engine never did for you: make the invariant part of what it watches for conflicts. And here's the uncomfortable preview of everything that follows: every way of doing that ends up serializing these writes somehow, and every one sends you a bill.

the rule    Σ(team.seats) ≤ 100              ← the engine never watches this

the move    materialize it into one doc every writer must also touch:
            license = { usedSeats: 90, maxSeats: 100 }
            → the invisible skew becomes a write-write conflict it CAN catch
              (doing it right — and what it costs — is the rest of this series)

The pattern underneath

Strip it back and the lesson is one sentence. Write skew isn't a flaw in transactions, and MongoDB isn't "wrong." Write skew is what happens when an invariant never becomes part of the database's conflict-detection model — a mismatch between the scope you validate (all the Team docs) and the scope the engine watches for conflicts (the one doc you write). The same read scope ≠ write scope gap from the guard above, all the way up.

Hold that sentence — it's the lens for everything that follows. Every cure from here is just a different answer to one question: how do I drag the invariant into something that serializes these writes? And not one of them is free.

Where this leads

So we reach for the reflex everyone tries first: a distributed lock. Freeze the world around the read and the write with Redis, and on paper you're finally serializable. In practice it's a minefield — latency, deadlocks, and a TTL dilemma with no good answer.

That's where things get ugly. Part 3.

Invariant-Driven Architecture: 20M transactions on a €80/mo Cloud VM.

Hugo Vantighem — Mon, 25 May 2026 06:27:00 +0000

📎 This is Part 2. Part 1 — Postgres-grade serializable at 20k ops/s on a laptop (don't try this at home) presented 20,000+ durable, invariant-validated transactions per second — on a MacBook Air M3, 8 cores, fan barely audible.

A laptop number is only half the story. The natural next test is whether the same architecture holds on a small, cheap public VM with network-attached storage and strict fsync(2) durability — three constraints that each, on its own, tend to move the bottleneck by an order of magnitude on most stacks. So that's what I ran.

The Setup

VM: Scaleway POP2-2C-8G — 2 vCPUs AMD EPYC 7543 @ 2.8 GHz, 8 GiB RAM. Yes — two vCPUs.
Disk: SBS Block storage, 100 GB, 15,000 provisioned IOPS. Network-attached, not local NVMe.
OS: Ubuntu 22.04, kernel 5.15. Linux strict fsync(2) — every commit hits the SSD for real, no Apple-style controller-cache shortcut.
Software: the exact same codebase that ran on the M3, with NATS + Mongo + Mongo Express as the side stack in Docker.
Price: ~€80/month order-of-magnitude — ~€54 of compute (POP2-2C-8G at €0.0735/h) plus ~€20–25 of SBS volume with provisioned IOPS. Hourly that's €0.11.

Two names, one stack

Two terms surface across this series. They sit at different layers, so it's worth pinning them now.

Invariant-Driven Architecture (IDA) — the design philosophy. The system, end to end (ingress, sequencer, system, storage), is engineered around a single obsession: validating and enforcing business invariants on every commit, with no compromise on throughput. It's a DDD-based philosophy.
Atomic State Platform — the concrete implementation. The software we're benchmarking right here — that just put down 33,091 sustained items per second on a €80/mo Scaleway VM.

IDA is how. Atomic State is what. The €80/mo VM is where. The rest of this post is how much.

1. The Visual Punch — 10 minutes non-stop, 20 million items 📈

The first test is the one that closes the "but does it sustain?" question forever.

BENCH_DURATION=10m BATCH_SIZE=1000 make cloud-bench-broker

Result, over 600 consecutive seconds on the POP2:

Metric	Value
Throughput sustained	33,091 items/s
Total items committed	19,855,000
WAL bytes written	7.6 GB
p99 round-trip	71 ms
Integrity audit	✅ INTEGRITY_OK (1,000 aggregates)
Durability	FSYNC-ON (Linux strict)

Yes, that's higher than the laptop — and on network-attached storage, not local NVMe. SBS is block storage over the data-center fabric; every fsync round-trips to the SBS backend before it returns. Measured fsync latency on this volume: 2.0 ms (vs 130 µs on the laptop's local NVMe — 15× slower per call). The 33,091 items/s holds despite the network disk, not because of a fast one. Pebble's group commit amortises that 2 ms across the whole batch — roughly 2 µs of effective fsync cost per item. That ratio is the architectural lever.

More on why the cloud beats the laptop on this same scenario in §4. For now, the headline isn't the number. The headline is the steadiness.

I sampled the engine's /metrics and the host's resource counters every 15 s through the entire run. Three things every senior engineer should care about:

pebble_health.l0_files stays in [0, 6] the whole 10 minutes. Never higher. Compaction kicks in the moment L0 hits 6, completes before the next batch needs the slot.
compactions_in_progress oscillates 0 ↔ 1 — Pebble is keeping up in real time, never queueing.
estimated_debt_bytes never crosses 100 MB — less than 1% of the WAL written.

At this scale — 34 MB/s sustained ingestion, ~20 GB of raw data committed — a mis-tuned LSM-tree triggers a Write Stall: the engine has to pause writes while compaction catches up, and the curve falls off a cliff. We see none of that. The puts counter climbs linearly from 0 to 19.85 M across the whole 10 minutes.

CPU%, L0 files and debt_MB never drift outside the bands shown above during the active 10-minute window.

[preflight] measured: lat_avg=2 051 µs iops=487
[preflight] Mac ref:  lat_avg=131 µs   iops=7 568 (M-series NVMe, same fio command)
[preflight] gate:     lat_avg <= 5 000 µs (override via SCW_FSYNC_MAX_LAT_US)
✅ PASS - instance is fsync-fast enough for a meaningful bench.

That's what "industrial-grade" looks like on a VM that costs €0.11/h.

2. The Ceiling Demo — batch=1000 vs batch=2000

Now the test that tells us where the wall is. Same VM, same 1-minute window, double the batch:

+2.8% throughput for +59% tail latency. Past 1,000 items per batch, more batching is just queueing. We've squeezed the disk dry.

Let's do the math on what the bottleneck is. The cloud-up preflight measured 2.0 ms average fsync latency on this SBS volume (fio --rw=randwrite --bs=4k --direct=1 --sync=1, the same command everywhere). At 33k items/s and one fsync per batch, that's 60 ms of fsync wall-time per second — 6% of the budget. Pebble's group commit already amortises fsync across the whole batch.

The remaining 94% of wall-time is CPU:

HTTP/JSON serialization on the producer
1,000 invariant evaluations per chunk
Pebble's batch building + commit

The network-attached disk is no longer the wall. Two 2.8 GHz AMD EPYC vCPUs at sustained 70% CPU are.

The fact that doubling the batch buys 2.8% is the experimental proof: there's no more disk to amortise, only CPU to share.

The full numbers, side by side

Full side-by-side numbers for batch=1000 vs batch=2000.

Read the bold rows together: we halved the number of fsyncs (33.3 → 17.1 per second) and throughput barely moved (+2.8%). If the disk were the wall, cutting fsyncs in half would have bought a lot more. It didn't — because CPU holds a flat ~70% plateau across both runs. That's the ceiling, in two numbers.

Engine + system samples, every 5 s (abridged — start / mid / end). Same CPU band, same L0 transient peak (10), and acks (= fsyncs) running at exactly half the rate on bs=2000 for the same puts — one fsync per batch, batches twice as big, no throughput dividend.

3. The Proof by Absurdity — 64 Workers on 2 Cores

This is the test every junior engineer expects to help throughput, and every senior engineer expects to kill it. The reflex: "if the engine is slow, scale out the producer."

make cloud-bench-perf-dense   # 64 workers, batch=100

Same VM. Same disk. Same system. The only change is the producer pattern.

Read that twice. CPU usage drops from 70% to 30% — and throughput collapses 5.5×.

That asymmetry is the signature of context-switching hell. The Linux scheduler spends its cycles swapping 64 producer goroutines + 64 HTTP request handlers + the system + 3 Docker containers across 2 physical cores. The real work-per-cycle drops; the cores look idle because they spend their time saving and restoring register state. The engine's single-writer queue becomes the rendezvous point — workers pile up — p99 explodes to 1.7 seconds.

This is the design principle of the engine stated as a measurement:

The engine is single-writer by design — one writer to Pebble, no lock contention.
The ingress must batch upstream of the engine to amortise fsync.
More producer threads = less throughput on a CPU-constrained host. Mathematically, not ideologically.

4. The Mac Parallel — More Cores Buy More Headroom

The same three scenarios, on the MacBook Air M3 from Part 1 (8 cores, 16 GB):

Three readings:

(a) On a single-worker pattern, the cloud Linux wins. Mac fsync(2) is ~15× faster than the cloud SBS per call (~130 µs vs ~2 ms), but at batch=1000 the per-batch fsync is amortised over 1,000 items — and the rest of the pipeline (HTTP serialization, internal sequencing, scheduler) now dominates.

(b) When the workload has CPU work to spare and concurrency is contained, the extra cores cash in. Going from batch=1000 to batch=2000 adds compute per batch but releases parallelism inside the engine (more items concurrently invariant-checked by the system). The Mac has 6 extra cores to spend on it, so its throughput climbs +54% (23,755 → 36,549). The cloud, pinned at 2 vCPUs, gains only +2.8% on the identical change — it has no spare core to convert the extra parallelism into work.

(c) The Mac does not flinch at 64 workers — it has the cores to absorb them. This is the exact scenario where the 2-vCPU cloud VM collapsed to 5,992 (§3). The 8-core Mac runs the identical 64-worker, batch=100, payload=0 B workload at 43,392 items/s — 7.2× the cloud, and above its own single-worker broker run (23,755). Context-switching only becomes hell when threads vastly outnumber cores; with 8 cores the scheduler keeps up and the engine's single writer stays fed. The perf-dense collapse was never about the workload — it was about core count.

The hierarchy of constraints is universal, regardless of OS, disk brand, or vendor SKU:

batch size > producer concurrency > raw fsync speed

Match those three to your hardware and your real ingress pattern, and the throughput follows. Get them wrong and a top-end laptop loses to a €80/month VM — or wins against one — depending on the day.

The Numbers, at a Glance

€80/month is the order of magnitude — ~€54 of compute, ~€20–25 of provisioned-IOPS SBS volume. Hourly: €0.11. The whole bench session that produced the cloud rows of this table cost ~€0.05 of cloud time — at this scale, validating an architecture decision on a representative VM is essentially free.

How the runs were captured

Each row above came from the same harness: a fresh VM with a clean Pebble, system + Mongo projection started as systemd units, NATS + Mongo + Mongo Express brought up in Docker, and the engine's /metrics endpoint sampled every 5–15 seconds during the run. An fio pre-flight (--rw=randwrite --bs=4k --direct=1 --sync=1, 5 s) gates the run on a configurable fsync latency threshold; on this VM it measured 2,051 µs average, well under the 5 ms gate. Every number in the tables above is either a direct read from /metrics, a count from the bench JSON output, or a delta between consecutive samples — nothing synthetic, no extrapolation.

Conclusion

20 million durable, invariant-validated transactions in 10 minutes, on a public-cloud VM that costs less per month than a SaaS subscription. Every run ends with INTEGRITY_OK.

What Atomic State Platform does on the M3 in Part 1, it does unchanged on a €80/mo Linux VM:

The single-writer engine means the disk stops being the wall as soon as you batch upstream.
Smaller, slower-fsync CPUs reach the same throughput envelope on cheap cloud as a powerful laptop — provided the producer pattern is cooperative.
Bigger machines buy headroom for concurrency, not raw throughput.

You don't need a 64-core server. You don't need an NVMe array. You don't need a datacenter rack. You need the right pattern, applied to the right SKU.

Part 3 unpacks the deeper claim: how Invariant-Driven Architecture lets Atomic State Platform sidestep the classical database stack outright — no Postgres, no Redis, no event-sourcing scaffolding. Just a system with a single fsync per batch, doing what nothing else does on a €80/mo box.

Read Modify Write Is Where NoSQL Concurrency Bugs Begin.

Hugo Vantighem — Sun, 24 May 2026 12:11:46 +0000

Part 1 of 3 — the single-document case.

There's a class of bug that every backend engineer ships at least once, usually
without noticing for months. It hides inside the most innocent-looking operation:
read a document, decide something, write it back.

Take a concrete invariant: a team can hold at most 10 seats. To add a seat you
read the team document, count the seats, check count < 10, and write. A textbook
Read → Modify → Write.

Now run it twice at the same instant. Request A reads count = 9, decides "9 < 10,
fine", and writes 10. Request B, a millisecond apart, also read count = 9,
decided "fine", and writes 10. You now have a team that thinks it has 10 seats but
actually granted 11. Neither request did anything wrong on its own. One write
silently erased the premise of the other. This is a lost update, and it's the
core anomaly of the single-document case.

T0   A reads count = 9
T1   B reads count = 9
T2   A writes count = 10   ("9 < 10, fine")
T3   B writes count = 10   ("9 < 10, fine")

Reality:        11 seats granted
Database state: 10
Invariant:      violated, silently

Here's what teams actually reach for, and exactly what each option leaves on the
table.

The fat aggregate (atomic operators)

If you can express the whole mutation as a single atomic operator — $inc,
$push with $slice, or a conditional findAndModify — MongoDB applies it
atomically on the document. There's no read-then-write window, so no lost update.
For invariants that fit a single atomic expression, this is genuinely the right
tool, and you should reach for it first.

The catch: not every invariant fits. The moment your check needs branching ("if
the plan is free and count ≥ 5, reject") you're back to reading, deciding in
application code, and writing — and the window reopens. Embedding related data is
a perfectly good modeling choice; the trap is different. It's the temptation to keep
stretching one document's consistency boundary — folding in unrelated rules just
to keep the write atomic — which is exactly how you end up with 16 MB documents and a saturated network.

Anomaly status: ✅ lost update handled — for the subset of rules expressible as
one atomic op.

The pessimistic lock (Redis)

Grab a distributed lock before the read, release after the write. It works — but
for a single document it's a sledgehammer. You've added a network round-trip, a
brand-new failure mode (the lock service), and a whole class of distributed
coordination failures — lease expiry, lock drift, fencing, split-brain — all to
guard one document the database could have guarded itself.

Anomaly status: ✅ everything — at the cost of latency and distributed coordination
failures. (Part 3 is dedicated to why that bill is steep.)

Optimistic locking (a version field)

Carry a version on the document. Read it, run your logic, then write with a
guard: findAndModify({_id, version: v}, {$set: {...}, $inc: {version: 1}}). If
anyone wrote in between, version moved, your guard matches nothing, and you
retry. This is the clean default for single-document RMW that doesn't fit an
atomic operator — it kills lost update with no external system.

The catch: under contention it's a retry machine. The more concurrent writers, the
more losers re-run their logic, burning CPU and tail latency.

Anomaly status: ✅ lost update — at the cost of app-side retries.

Pray

Bet that two requests never touch the same document in the same millisecond. They
will. Anomaly status: ❌ lost update, in production, at 3 a.m.

The point

For a single document, you're actually well served: atomic operators or optimistic
locking close the gap cleanly, without external machinery. The single-document
case is the easy one.

The real pain begins the instant your invariant spans two documents — a
workspace budget gating a user debit, for example. There, optimistic locking stops
being sufficient: it still guards each document on its own, but it can no longer
guarantee an invariant that lives between them. And a nastier anomaly walks in —
the database stays perfectly "consistent" while your business invariant quietly
dies.

Welcome to write skew. That's part 2.

Postgres-grade Serializable at 20k+ ops/s — on a laptop. Don’t try this at home.

Hugo Vantighem — Sat, 23 May 2026 17:14:52 +0000

They didn't know it was impossible, so they did it. — Mark Twain

In the software industry, we've been raised with a dogma: you must choose between Massive Performance (NoSQL, eventual consistency) and Domain Rigor (SQL, strong consistency, serializable).

We are told that locks, latencies, and ACID properties are the natural enemies of speed. That if you want to scale, you have to let go of your business invariants.

I decided to test another hypothesis. And I broke the myth.

The Result: 20,000+ Validated Transactions per Second

This isn't a "fire and forget" ingestion log.

This isn't a volatile cache experiment.

What you see here is Business Transaction Durability:

Invariants validated — every business rule is checked before commit.
State persisted — every change is durably written to disk.
Strong Consistency — Serializable-level isolation.

At 20,000+ ops/s, we are not just talking about speed. We are talking about the ability to maintain absolute domain integrity under massive load.

And the kicker: this is running on a MacBook Air M3 — 8 cores, 16 GB of RAM, the same machine I write the code on. No 64-core server. No NVMe array. No datacenter rack. One laptop, fan barely audible, doing the work of a small cluster.

Why General-Purpose Databases Hit a Ceiling

Most databases are built for general cases. They treat every row the same way because they don't know your business.

This "Domain Ignorance" leads to generic row locks, MVCC bookkeeping, cross-table coordination, and massive overhead — costs you pay on every single transaction, whether your domain needs them or not.

Not Magic — Discipline

For the skeptics: this isn't sorcery. It's discipline applied to the right layer — designing the system so the hardware does exactly what it's good at, and nothing else.

I'm not reinventing the storage wheel. The foundation is Pebble, the same proven LSM-tree engine that powers CockroachDB. But the engine is just the floor. The real lever is the orchestration of the domain logic on top of it — and that's what Part 2 puts a name on.

A Note on the Benchmark Scope

I know what you're thinking. "20k+ ops/s? That must be an internal memory trick."

It isn't. To ensure these numbers reflect real-world usage, the benchmark covers the entire lifecycle of a business transaction:

Client-side serialization — the payload starts from the app.
Local communication — end-to-end roundtrip.
Server-side deserialization & parsing.
Business Invariants validation.
Disk persistence with full durability guarantees — fsync on every commit.

The workload: batch=1000, payload=1KB, single-node, single laptop. Here's the run, with the system-level disk stats captured live during the bench:

[23755.87 items/s] | items=1424000 | batch=1000 | payload=1KB | durability=FSYNC-ON

Live capture during the bench (batch=1000, 1KB, fsync ON). Disk on fire, CPU bored.

Two things jump out of that stats panel — and together they're the whole point:

The disk is screaming. Sustained 100–200 MB/s with the ⚡ markers firing almost every second. This is real fsync'd traffic hitting the SSD, not a memory cache pretending to be durable. If you pulled the power cord mid-run, every committed transaction would still be there on reboot.
The CPU is bored (~18% on an 8-core M3). The compute is idle while the disk pegs out — that asymmetry is the whole story.

And this isn't the ceiling. With bigger batches the same laptop pushes further; even at batch=1, it doesn't fall off a cliff. The full envelope is Part 2.

What's Next?

This is just Part 1. In a few days, Part 2 finishes the picture and lands the real punchline: business rules aren't a tax on performance — they're the contract that lets the machine fly. And the whole thing runs on hardware your team could expense, not a cloud bill that needs board approval.

Stay tuned. The era of the "Impossible Trade-off" is over.