Where MongoDB puts the writes it took back

A MongoDB replica set can tell your app a write succeeded and then lose it. That isn't a bug, it's what the write concern you choose allows. I wanted to see it happen, count it, and find out where the lost writes go, because MongoDB doesn't just throw them away. It writes them to a folder on the old primary's disk that, as far as I can tell, most people never open.

So I ran a three-member replica set in Docker, wrote to it from eight threads, and broke the primary two different ways: killing it, and cutting it off from the other two members while the app could still reach it.

The setup

• MongoDB 8.3.11 (the mongo:8 image), three members in one replica set, each in its own container on Docker Desktop on an ARM Mac
• the client in a fourth container, Python with pymongo 4.19
• each writer thread inserts tiny documents (_id and a timestamp) as fast as it can and logs every insert that was acknowledged
• after each test, I compare the acknowledged ids against what the replica set actually has

On this version the default write concern is already majority. db.adminCommand({getDefaultRWConcern: 1}) returned {w: "majority"}. So w:1 is something you have to ask for. People do ask for it, because it's faster. Before any failure, the median insert took 0.28 ms with w:1, 1.12 ms with w:1, j:true and 1.34 ms with majority.

Killing the primary

Eight threads write for 20 seconds. Five seconds in, I SIGKILL the primary container, then start it again once the run ends and let it rejoin. Ten runs per setting:

Write concern	Acknowledged writes	Lost	Runs with a loss	Errored, but saved anyway
w:1	480,466	5	3 of 10	0
w:1, j:true	420,104	0	0 of 10	27
majority	387,067	0	0 of 10	30

Five lost writes in almost half a million is a small number. But they were writes the app had been told were saved. They were acknowledged 21 to 23 ms after my script sent the kill, so they were the last few writes before the process died.

None of the five ended up in a rollback file, which is what pointed me at the cause. A w:1 write is acknowledged once it's in the primary's memory, not once it's in the journal on disk, and the journal is flushed every 100 ms by default. Kill the process in between and the write is simply gone, the restarted node never had it, so there's nothing to roll back. With j:true the acknowledgement waits for the journal, and nothing was lost in ten runs. Ten runs don't prove that, but it fits.

The last column is the one I didn't expect. With j:true and with majority, every run ended with a few writes that had returned an error to the app and were in the database anyway. A write that fails during a failover isn't a write that didn't happen. It's a write whose outcome you don't know.

I also ran one kill with a single writer thread for each setting first. Neither lost anything. With one thread the secondaries kept up within a fraction of a millisecond, and there was never anything unreplicated to lose.

Cutting the primary off

A dead primary is the easy case. The harder one is a primary that's still alive and still reachable by your app, but can't reach the other members. I set this up with two Docker networks. The members replicate over one, the client talks to them over the other, and at the five-second mark I took the primary off the replication network and the other two off the client network.

Two writers ran this time, four threads each. One wrote straight to the old primary. The other used the normal replica set connection string, the way an app would.

With w:1, five runs:

Run	Old primary stepped down after	Acknowledged writes lost	Documents in the rollback file
1	10.20 s	66,990	66,990
2	10.32 s	62,743	62,743
3	10.32 s	55,569	55,569
4	10.23 s	67,225	67,225
5	10.68 s	57,728	57,728

For about ten seconds the old primary kept accepting writes it couldn't replicate. Then it noticed it had lost the majority and stepped down. Meanwhile the other two had elected a new primary and carried on without those writes. When I reconnected the network, the old primary found it had history nobody else had, and rolled it back. Every write acknowledged in those ten seconds was lost, and the number in the rollback file matched the number of lost writes exactly, every time.

The writer using the replica set connection string lost as many writes as the one connected directly, about half of each total. It could still reach the old primary, the old primary still said it was primary, so that's where its writes went. And it saw no errors at all. When the old primary stepped down, the driver moved to the new one and carried on. Its log has nothing in it to tell you anything happened.

I went looking for a window where both sides accepted writes, and found the opposite. In the first run's server logs the old primary stepped down 10.03 s after the cut and the new one was elected at 11.35 s. For 1.3 seconds there was no primary at all.

The same test with majority, three runs: no acknowledged write was lost. The old primary still applied writes during the cut, but it couldn't get them acknowledged, so the app got errors, InterruptedDueToReplStateChange after about 3.9 seconds or a network timeout at 6. Those writes were rolled back too, 8, 13 and 13 of them, and every one of them was a write that had already returned an error.

The rollback folder

After a rollback, the old primary's data directory has a new folder:

/data/db/rollback/<collection UUID>/removed.2026-10-28T07-33-00.1.bson
/data/db/rollback/local.oplog.rs/removed.2026-10-28T07-33-00.0.bson

The first file has every rolled-back document in full. The second has the oplog entries that were removed, including session ids and transaction numbers. They're plain BSON:

$ bsondump --quiet removed.2026-10-28T07-33-00.1.bson | head -1
{"_id":"old-33372","t":{"$numberDouble":"1.793172757384536E+09"}}

The folder is named after the collection's UUID, not its name, so you need db.getCollectionInfos() to work out which collection it belongs to. In my runs it didn't match anything, because my script dropped and recreated the collection before each run. On a real database it would.

You can put the documents back. mongorestore into a separate collection brought back all 57,728 documents from run 5. Deciding which of them should go back into the real collection is the hard part, since the new primary has had writes of its own since then.

Nothing tells you any of this happened. The old primary's log has a "Rollback summary" line and the folder appears on its disk. The app isn't told, and with the replica set connection string, it didn't even see an error.

What I got wrong

My first kill runs used one writer thread and lost nothing, with w:1 too. I nearly wrote that w:1 survives a crash fine. It only looked that way because one thread never got ahead of the secondaries. With eight threads, three runs in ten lost writes.

The first partition run reported that the replica-set writer had failed every single insert. It hadn't. Both writers were generating the same _id values into the same collection, and the second one was hitting duplicate key errors. I gave each writer its own prefix and ran it again.

I also counted rollback files by listing the whole folder at first, which includes files from earlier runs. The numbers above come from comparing the folder before and after each run.

Caveats

All three members ran on one laptop, so replication was much faster than it would be between real machines. Over a real network, secondaries fall further behind, and I'd expect a kill to lose more w:1 writes, not fewer. The partition was made with Docker networks and lasted 25 seconds, and ten seconds of lost writes is roughly the default election timeout. The documents were tiny and I only tested inserts. The "errored, but saved" counts depend on how the driver retries, and I used pymongo with its defaults, retryable writes on.

What I'd take from it

• Leave the write concern at majority unless you've decided which writes you can afford to lose. w:1 didn't just lose writes in a crash, it lost ten seconds of them in a network split, with no error.
• A write that errors during a failover may still have happened. Make writes idempotent, or check before you retry.
• If you use w:1 anywhere, look at /data/db/rollback on your members after any failover, and alert on the "Rollback summary" log line.
• Rolled-back writes aren't deleted, they're saved as BSON and mongorestore can bring them back. Know where the folder is before you need it.