DEV Community

Zayd Mulani
Zayd Mulani

Posted on

I built a tool that pulls the power cord on your program (thousands of times)

write() returning doesn't mean your data is on disk. Even fsync() doesn't cover everything. A file system may persist writes out of order, tear them at block boundaries, save a file's new size without its data, and drop a new directory entry unless you fsync the directory. Code that passes every test can still lose data, or corrupt its state, when the power fails at the wrong moment.

These bugs are common in serious software. Pillai et al. (OSDI '14) found 60 of them across 11 applications, including databases and version-control systems. But the tool from that paper, ALICE, is unmaintained. So I built unsynced.

What it does

  1. Record what a program does to a directory: strace for any program on Linux, or a Recorder API for Rust code on any OS.
  2. Compile each operation into the pieces a file system can persist independently (directory entries, 4 KiB write chunks, size changes), each tagged with the fsync that makes it durable.
  3. Enumerate every on-disk state a crash could leave under a persistence model (the weakest POSIX allows, or ext4's default).
  4. Check each unique state with your checker. It gets the directory plus what the program had acknowledged (printed) before the crash.
  5. Explain. Each failure is minimized to the lost operations that cause it, classified, and turned into a fix.

    unsynced run --init seed/ \
    --check 'c=$(cat {dir}/cfg); [ "$c" = v2 ] || { ! grep -q saved {marks} && [ "$c" = v1 ]; }' \
    -- sh -c 'printf v2 > {dir}/tmp && mv {dir}/tmp {dir}/cfg && echo saved'

    [1] reordering (7 failing states)
    lost #1 write tmp [0..2)
    but persisted #2 rename tmp -> cfg
    fix fsync tmp before rename tmp -> cfg (op #2); the rename reached disk before the data it points to

    [2] unsynced (1 failing states)
    lost #2 rename tmp -> cfg
    fix fsync the directory . after rename tmp -> cfg and before acknowledging ("saved")

The SQLite test

To see whether the model was realistic, I ran SQLite under every journal mode and synchronous level, with a checker demanding integrity plus every acknowledged commit. unsynced knows nothing about SQLite:

mode unsynced SQLite docs
DELETE + FULL commit lost: journal unlink not dir-fsynced EXTRA exists to sync the dir after that unlink
DELETE + EXTRA clean ACID
WAL + NORMAL commit lost: WAL not fsynced before commit returns "might roll back following a power loss"
WAL + FULL clean ACID
OFF database disk image is malformed "might become corrupted"

Every row matches the docs, and CI re-runs this on every push.

Crashing the recovery

Recovery code writes to disk too. With --recover, each repair is traced and crashed at every step, then run again. The example in the repo is a log whose torn-tail repair rewrites the file: correct in every normal test, but it loses acknowledged data if the power dies mid-repair. The same repair done with set_len survives.

Try it

cargo install unsynced
Enter fullscreen mode Exit fullscreen mode

Repo, model spec and limitations: https://github.com/zaydmulani09/unsynced

Top comments (0)