My database claimed crash-safety. `kill -9` proved nothing.

I have a small document database called grange. Its README said "crash-safe by construction", and there was a harness to back it up: kill the writer with kill -9 mid-flight, reopen, assert the database holds exactly a committed prefix. Five rounds, every build, green.

That harness proves less than it looks like it proves.

What kill -9 actually tests

When you kill -9 a process, the kernel reclaims it — but everything the process wrote with write(2) is already in the page cache, and the kernel flushes it to disk afterwards on its own schedule. The data survives because the operating system survived.

A power cut is different. So is a kernel panic. There, the page cache goes with the machine, and anything not yet on the platter is gone.

My database called write_file and returned success. Nothing anywhere called fsync. So a commit that had been acknowledged — the client got its {"ok":true} — could vanish. The harness could never have caught it, because the failure mode it simulates is the one the page cache is immune to.

The README even said so, in a line I'd stopped reading:

Durability is process-crash-exact (proven by make crash), OS-crash best-effort.

That sentence is true and completely insufficient. "Best-effort" is doing a lot of work there.

Two fsyncs, not one

The fix is fsync, and the part that's easy to get wrong is that one call isn't enough.

write_file(path, data)
fsync(path)        // the file's CONTENTS are durable
fsync(dirname)     // the file's EXISTENCE is durable

Creating a file is a modification of its directory. You can fsync a file perfectly and still lose it after a crash, because the directory entry naming it never reached the disk. The second fsync is the one people skip.

A directory has to be opened O_RDONLY for this — opening it for writing fails with EISDIR.

Testing something invisible

Here's the awkward part: fsync is invisible to behavioural tests. A build that silently dropped every fsync call would pass my entire test suite. Same answers, same recovery, same everything — right up until the power goes out.

So the test doesn't check behaviour. It checks syscalls:

strace -f -e trace=openat,fsync ./db put ...

then asserts the ordering: the chunk is fsynced, then its directory. And on the disk-resident path, that every data page is fsynced before the manifest that references it — because "write the manifest last" is only meaningful if that ordering is real on the platter, and the kernel is free to reorder write() calls on the way there.

The third check is the one I'd encourage you to steal: the opt-out (FSYNC=0) must issue zero fsyncs. That doubles as the harness's own negative control. A test that can't fail is decoration, and a test asserting the presence of something invisible needs to prove it can detect the absence.

What it costs

Measured, on a commit:

single-document commits	~1.9 ms each — about 31% slower
a 10,000-document batch	21 ms vs 23 ms — noise

That shape is why it's on by default. A batch is one commit, so durability amortises to roughly nothing on the workload that actually moves volume. The per-commit case pays, and that's the honest price of the guarantee.

What I still can't claim

The device might be lying. Consumer drives with volatile write caches can acknowledge an fsync that's still in flight, and no test running on that machine can tell. So the docs now say: fsync is called, the ordering is asserted, and whether your hardware honours it is not something this project can prove.

That felt more useful than upgrading the marketing adjective.

The general lesson

The harness wasn't wrong. It tested a real failure mode, correctly, and it still passes. It just wasn't testing the failure mode the word "crash-safe" makes people imagine — and the gap between those two things sat in my README for months, written down, in a sentence I'd read so many times I stopped seeing it.

If you have a durability claim, go and check what your test kills. kill -9 and "pull the plug" are not the same experiment.

---

grange is MIT, a single static Linux binary, and usable either as a server or compiled into your own program: github.com/javimosch/grange. v0.11.0 also brings ordered queries, keyset pagination and field projection.

Comments

[Mads Hansen]

Excellent distinction between process-crash recovery and durability. The next edge I’d test is the full atomic-replacement protocol: write a temp file, fsync it, rename it over the old file, then fsync the parent directory. If a manifest points to multiple data files, every referenced file must be durable before the manifest becomes durable; otherwise recovery can discover a perfectly synced manifest that names missing data.

strace is a strong structural test, especially with the negative control. I’d pair it with a storage fault harness (for example a VM/block device that can drop or reorder writes) and run crash points after every persistence step across the filesystems you claim to support. That catches assumptions hidden by one kernel/filesystem combination. The documentation phrasing is the real win: specify the acknowledged-commit boundary, syscall/order guarantees, filesystem scope, and hardware caveat instead of collapsing all of that into “crash-safe.”