write() returning doesn't mean your data is on disk. Even fsync() doesn't cover everything. A file system may persist writes out of order, tear them at block boundaries, save a file's new size without its data, and drop a new directory entry unless you fsync the directory. Code that passes every test can still lose data, or corrupt its state, when the power fails at the wrong moment.
These bugs are common in serious software. Pillai et al. (OSDI '14) found 60 of them across 11 applications, including databases and version-control systems. But the tool from that paper, ALICE, is unmaintained. So I built unsynced.
What it does
-
Record what a program does to a directory:
stracefor any program on Linux, or aRecorderAPI for Rust code on any OS. - Compile each operation into the pieces a file system can persist independently (directory entries, 4 KiB write chunks, size changes), each tagged with the fsync that makes it durable.
- Enumerate every on-disk state a crash could leave under a persistence model (the weakest POSIX allows, or ext4's default).
- Check each unique state with your checker. It gets the directory plus what the program had acknowledged (printed) before the crash.
-
Explain. Each failure is minimized to the lost operations that cause it, classified, and turned into a fix.
unsynced run --init seed/ \
--check 'c=$(cat {dir}/cfg); [ "$c" = v2 ] || { ! grep -q saved {marks} && [ "$c" = v1 ]; }' \
-- sh -c 'printf v2 > {dir}/tmp && mv {dir}/tmp {dir}/cfg && echo saved'[1] reordering (7 failing states)
lost #1 write tmp [0..2)
but persisted #2 rename tmp -> cfg
fix fsynctmpbeforerename tmp -> cfg(op #2); the rename reached disk before the data it points to[2] unsynced (1 failing states)
lost #2 rename tmp -> cfg
fix fsync the directory.afterrename tmp -> cfgand before acknowledging ("saved")
The SQLite test
To see whether the model was realistic, I ran SQLite under every journal mode and synchronous level, with a checker demanding integrity plus every acknowledged commit. unsynced knows nothing about SQLite:
| mode | unsynced | SQLite docs |
|---|---|---|
| DELETE + FULL | commit lost: journal unlink not dir-fsynced | EXTRA exists to sync the dir after that unlink |
| DELETE + EXTRA | clean | ACID |
| WAL + NORMAL | commit lost: WAL not fsynced before commit returns | "might roll back following a power loss" |
| WAL + FULL | clean | ACID |
| OFF | database disk image is malformed |
"might become corrupted" |
Every row matches the docs, and CI re-runs this on every push.
Crashing the recovery
Recovery code writes to disk too. With --recover, each repair is traced and crashed at every step, then run again. The example in the repo is a log whose torn-tail repair rewrites the file: correct in every normal test, but it loses acknowledged data if the power dies mid-repair. The same repair done with set_len survives.
Try it
cargo install unsynced
Repo, model spec and limitations: https://github.com/zaydmulani09/unsynced
Top comments (0)