DEV Community

Semurg
Semurg

Posted on Originally published at one.semurg.io

We kill -9 our database 21 times in 140 seconds, then read back every write. Here is what comes back

A database earns trust by what it gives back after something goes wrong. So for Semurg we break a running node the ways real machines break, then read back every write the node acknowledged. These results are from the release published on 29 October 2026 (r11.138), on one bare-metal AMD EPYC server.

A killed process

32 writers kept writing while we killed the node with kill -9 21 times in 140 seconds, restarting it each time. Afterwards we read back all 32.2 million acknowledged writes, each at its newest acknowledged version. None was lost.

A full disk

We kept writing until the data volume filled. As it neared full, the node refused new writes by name (HTTP 507, "insufficient storage") and stayed healthy, and all 2,183,488 writes it had acknowledged read back. Then we grew the volume while the node kept running: writes were accepted again with no restart, and all 6,190,464 acknowledged writes read back.

A slow disk

We throttled the disk to 200 MB/s, then 40 MB/s, then 8 MB/s, under write load. The node got slower and never wrong: every acknowledged write read back at every level.

A frozen process

We paused the node 5 times for 15 seconds each under load, and all 15.2 million acknowledged writes read back. 80 client calls timed out during the pauses. None of those was an acknowledged write that went missing: a call that times out was never acknowledged, so the client sends it again, and every call sent again was stored.

A power cut

We replayed sudden power loss at the block device, on a datacenter drive with power-loss protection, 21 cuts in each of two data shapes. Everything acknowledged before the final seconds survived every cut, and nothing that read back was ever damaged. Writes acknowledged in the last seconds before a cut can be lost on ext4: at most 9.4 seconds in one shape and 4.7 seconds in the other, inside the 10-second bound we test against. Our goal there is zero, and we publish the number until we get there.

Why we test this way

The rule we hold ourselves to is simple: when the node cannot keep a write, it refuses that write at the moment you send it, by name, so nothing is accepted now and lost later. Every result above is on our durability page with the release it was measured on, and the tests run again on new releases.

What failure do you test your own database against before you trust it with data? A killed process, a full disk, a power cut, something nastier? I would like to hear what has bitten you.

Read more on our blog: https://one.semurg.io/blog/what-a-semurg-node-gives-back

This post was drafted with AI assistance and checked line by line against our published durability results.

Top comments (0)