"The write was acknowledged" means less than you think

I spent years operating databases for payment platforms, where a lost write isn't a bug, it's a regulatory conversation. That environment teaches you to read the phrase "the write was acknowledged" the way a lawyer reads a contract. Acknowledged by whom? Durable against what?

In MongoDB, that one sentence has at least four different meanings, controlled by two settings most applications leave at defaults without ever deciding to.

The four contracts

// 1. w:1, the primary has it in memory
db.payments.insertOne(doc, { writeConcern: { w: 1 } })

// 2. w:1, j:true, the primary has it in its journal on disk
db.payments.insertOne(doc, { writeConcern: { w: 1, j: true } })

// 3. w:"majority", a majority of the replica set has it
db.payments.insertOne(doc, { writeConcern: { w: "majority" } })

// 4. w:"majority", j:true, a majority has it journaled

Each level survives a different failure.

• w:1 survives nothing interesting. If the primary's mongod crashes before the next journal flush, the write is gone, while your application already told the user "success."
• w:1, j:true survives a process crash on the primary. It does not survive the primary failing over.
• w:"majority" survives failover. This is the one that matters, and here's why.

Where acknowledged writes go to die

A replica set election isn't polite. If the primary accepts writes at w:1, then loses connectivity before replicating them, a secondary gets elected and moves on. When the old primary rejoins, it discovers history diverged. It has writes the new primary never saw.

Those writes get rolled back. MongoDB doesn't delete them silently, it writes them to BSON files in the rollback directory, where, in my experience, they are examined by precisely no one until an auditor asks a question.

w:"majority" closes this hole. The write isn't acknowledged until enough nodes have it that any electable primary must have it too. It cannot be elected away.

The read side of the same contract

Durable writes with careless reads is half a system. Two settings complete it:

// Don't show me data that could still be rolled back
db.payments.find({...}).readConcern("majority")

// My own session should see its own writes, in order
const session = client.startSession({ causalConsistency: true })

And on the driver, retryWrites=true (default in modern drivers) makes the failover window survivable. A write interrupted by an election is retried exactly once against the new primary, safely, because every write carries a unique transaction number.

Which is also why "just wrap it in a retry loop yourself" is worse than the built in one. Your loop can double apply. The driver's cannot.

What I actually deploy

Tiering, not one global setting.

Data	Write concern	Why
Payments, ledger, anything auditable	w:"majority"	Rollback is not an acceptable word here
User profile updates	w:"majority"	Cheap insurance, users notice lost edits
High volume telemetry, click events	w:1	Losing two seconds of events in a rare failover is a fair trade for throughput

The latency cost of majority is one replication round trip, single digit milliseconds inside a region. I've watched teams run payments at w:1 to save five milliseconds, which is a way of saying they priced a lost financial record at five milliseconds.

The lab

Start a 3 node replica set locally (mongod --replSet three times, or docker), then:

// terminal 1: insert at w:1 in a loop
// terminal 2: rs.stepDown() on the primary mid-loop
// then check the old primary's rollback directory

ls /data/rs1/rollback/

Run it. Seeing your own acknowledged write sitting in a rollback file is worth more than this whole post.

Defaults are decisions someone else made. For write concern, make your own, per collection, on purpose, written down.