How FaultBox helped me solve a storage corruption bug I couldn't reproduce

I was testing NodeDB-Lite and PageDB through a real memory-layer application built on top of them. NodeDB-Lite is the embedded form of NodeDB for local-first and in-process workloads, while PageDB is the encrypted page store underneath it.

That application was part of the test strategy. I did not want to validate the storage stack only through unit tests, fixtures, and controlled benchmarks. I wanted a real workload to keep using it, stress it, restart it, grow its data, and exercise the boundaries that isolated tests usually miss.

Then the store became corrupted.

The visible symptom was an authenticated-page read failure around an FTS path. A page that should have passed its AEAD authentication check did not. The application restarted, opened the same damaged store, hit the failure again, and fell into a restart loop.

The hard part was not proving that the store was corrupt. The hard part was reproducing how it became corrupt.

I could not reproduce it inside PageDB. I could not reproduce it through NodeDB-Lite. I could not even make the application produce it on demand. I could use the application normally for a while and eventually see the failure, but I did not have a deterministic sequence that caused it.

By the way, I still found bugs along the way.

Some were real. Some looked close enough to the corruption path that I thought I had finally found the root cause. I fixed them, rebuilt, ran the tests, and went back to dogfooding.

The corruption still came back.

At that point, I stopped asking:

Which storage bug looks plausible?

The real question was:

Where does it actually go wrong?

I kept testing the wrong shape of failure

My strongest theory was freed-page reuse, or something close to a use-after-free inside the store.

It was a reasonable theory. If a page had been released and then reused while another structure still referenced it, a later authenticated read could land on bytes that were valid somewhere else but invalid for the page the reader expected.

So I attacked that theory with tests. I stressed allocation and reuse. I pushed the FTS path. I exercised structural checks. I wrote increasingly aggressive single-process cases to catch the store corrupting itself.

Nothing reproduced the incident.

That should have weakened the theory sooner, but a plausible storage theory is difficult to let go when the only evidence you have is the final corrupted state.

The process that wrote the bad page might already be gone. The process that detected it only knew that authentication failed now. The logs could show nearby activity, but they did not preserve the store at the moment the failure was detected. A panic reporter was not enough either, because the corruption arrived as a returned error rather than a panic.

I had tests for behavior I understood. What I did not have was the evidence needed to understand this behavior.

I needed a black box

Then it clicked.

I needed a black box.

When the application detected corruption or crashed, I needed it to record what it still knew immediately, before another restart, cleanup path, or debugging attempt changed the state.

I did not start by designing a public crate. I built the smallest recorder I needed for this investigation.

The application would emit a structured capture at the detection site. The capture would keep the error and the surrounding report context, then preserve a copy of the corrupt store so I could inspect the exact artifact that had failed.

That first version did not have breadcrumbs. It did not have the shared-memory ring, native-crash monitor, or the rest of the public FaultBox feature set. PageDB's internal FaultBox instrumentation came later too.

It was minimal.

But it worked.

The next time the corruption surfaced, the recorder preserved the damaged store instead of leaving me with another disappearing error. I could run fsck against the captured copy and reproduce the bad state offline.

That alone changed the investigation. I was no longer trying to rebuild an incident from memory or force it into the wrong test. I had the store that had actually failed.

Across the incidents, one clue kept appearing: the store was already open.

That pointed to the boundary my single-process tests could never reproduce. Two processes had opened the same embedded store and were making allocator decisions without shared ownership.

The allocator was not independently corrupting itself inside one process. The ownership model around it was broken. Two processes both believed they owned the store, so a locally valid allocation decision in one process could invalidate what the other process believed.

The fix belonged at that boundary: one process, one store, enforced with a single-instance lock and a corrected application lifecycle.

I rebuilt a clean store and stressed the application again under that rule. The corruption stopped in this dogfood run.

There was another problem one layer up. Corruption discovered after the PageDB handle opened could lose its typed meaning while moving through NodeDB-Lite, collapse into a generic error, and feed the restart loop. The same investigation led me to preserve the typed corruption and add bounded recovery in Lite.

FaultBox did not diagnose all of that automatically. It did something more basic and, in this case, more useful: it kept the evidence that let me stop chasing the wrong failure shape.

From a debugging tool to a public crate

The minimal recorder had solved a problem I had already spent a long time chasing.

That was when I thought:

This is good. I should make it better, and I should make it public.

The first missing piece was obvious. The captured store showed me what had failed, but the report did not show the operations leading up to it. That pushed me to add the breadcrumb flight recorder.

The restart loop exposed another problem. Repeated captures could duplicate large store snapshots. A crash recorder that fills the disk while reporting the same failure has created a second incident. That led to fingerprint coalescing, occurrence counts, retention limits, and preserved-artifact reuse.

The root cause crossed a process boundary, which meant a detector process could have the wrong local history when another process caused the bad write. That led to the optional shared-memory ring, where breadcrumbs can be attached to the store rather than only to one process.

Redaction, owner-only report files, native-crash support, and the rest of the public API came from asking what this recorder would need before I could responsibly tell other people to use it.

The result is FaultBox: a Rust crate for recording forensic evidence and lead-up context at the detection site for panics, optional native crashes, explicit corruption, invariant violations, and returned errors that deserve a report. I published it because the need to preserve a failure before it disappears is not unique to my storage stack.

---

FaultBox complements tests, logs, and panic reporters

I do not want FaultBox to be mistaken for a testing strategy.

Tests are how a known behavior becomes enforceable. They are where a root cause should end up. After the two-process store ownership bug was understood, the important thing was not to keep admiring the preserved report. The important thing was to fix the boundary and cover the behavior that could be covered.

I still want the normal testing layers:

• Unit tests for local algorithms, error mapping, and invariants.
• Integration tests for the real dependency stack and lifecycle boundaries.
• Property and stress tests for input shapes and state transitions I would not enumerate by hand.
• Concurrency and fault-injection tests for interleavings and partial failure.

FaultBox starts where one of those tests does not yet contain the triggering state. It preserves what the real detector knew so the missing scenario can eventually become another test.

Logs are still useful because they explain routine activity and operator-visible progress. But logs are optimized for streams, not forensic bundles. They usually do not snapshot the corrupt file or directory, attach structured domain context, coalesce repeated occurrences of the same bug, or preserve a bounded report group for offline inspection.

Panic reporters are useful when the program panics. But storage engines and long-lived services often detect their worst failures as returned errors: checksum failure, authentication failure, impossible page kind, violated invariant, failed recovery precondition. No panic hook sees those unless the application turns the error into a panic, and turning storage corruption into a panic just to make a reporter notice it is the wrong abstraction.

FaultBox sits beside those tools. It records an explicit event when the code that detects the failure still knows what class of failure it is.

What FaultBox records

FaultBox v0.1.2 has one report model for several event kinds:

• Panic
• Optional NativeCrash
• Explicitly emitted Corruption
• Explicitly emitted InvariantViolation
• Explicitly emitted Error

The explicit cases are important. FaultBox does not detect corruption by itself. If a library reads a page and the authenticator fails, or an index sees a child kind that cannot be valid, that layer has to emit the capture. The layer that knows the invariant is the layer that should report it.

A capture may carry an error chain, domain context, a backtrace, breadcrumbs, build and process metadata, and a preserved file or directory artifact. Those fields are optional. Not every report contains every one of them.

Reports coalesce by fingerprint. With domain context, the fingerprint uses the project, event kind, domain kind, and stable grouping key. Without domain context, it falls back to a normalized message. The goal is to group the same bug, not every instance. Repeated reports update occurrence counts and first/latest timestamps instead of creating a new directory for every restart. Preserved artifacts can be reused when the same fingerprint repeats with the same artifact content.

Defaults are intentionally bounded: 128 breadcrumbs, a 256 MiB preserve cap, and best-effort retention of 64 groups or 2 GiB of recorded artifacts. All of those are configurable.

There are three ways to get lead-up context:

• Manual faultbox::breadcrumb! calls around meaningful operations.
• The optional tracing feature, where BreadcrumbLayer turns tracing events into breadcrumbs. It records INFO and above by default and supports target filters.
• The optional shared-ring feature, where an artifact-keyed shared-memory ring can merge breadcrumbs from other processes using the same store.

The shared ring exists because of the kind of bug described above. If process B detects a corrupt read, process B's local ring may not include the write sequence from process A. A shared ring cannot prove causality by itself, but it can preserve the breadcrumb history that a per-process recorder structurally cannot see.

When FaultBox is useful

FaultBox is a good fit for Rust software that detects serious failures it may not be able to reproduce immediately:

• Storage engines, embedded databases, sync engines, file-format readers, and anything that checks checksums, authentication tags, or structural invariants.
• Long-running services and desktop applications where the failed process may be gone before anyone can inspect it.
• Libraries that can detect their own corruption or invariant violations and want the host application to own report storage.
• Applications crossing FFI boundaries, if they choose to enable native-crash support for faults that Rust panic hooks cannot see.

It is probably not a good fit for no_std, bare-metal or firmware-style embedded targets, or applications whose only realistic failure mode is already handled well by a panic reporter. FaultBox is not a telemetry product. If you need a hosted event pipeline, dashboards, symbolication service, or fleet analytics, it is not that.

---

Install and initialize

FaultBox is available on crates.io, with its API documented on docs.rs. It requires Rust 1.88 or newer.

Add it to the binary that owns your application lifecycle:

[dependencies]
faultbox = { version = "=0.1.2", features = ["tracing"] }
tracing-subscriber = "0.3"

Only the binary initializes FaultBox. Libraries may emit breadcrumbs and captures, but they should not choose the report directory, redactor, panic hook, or native-crash policy.

use faultbox::{BasicRedactor, Config};

fn main() {
    let reports_dir = std::path::PathBuf::from("faultbox-reports");

    let initialized = faultbox::init(
        Config::new("myapp", env!("CARGO_PKG_VERSION"), reports_dir)
            .redactor(Box::new(BasicRedactor::new())),
    );

    if !initialized {
        eprintln!("faultbox was already initialized");
    }

    run_app();
}

fn run_app() {}

BasicRedactor is not the default. The default is NoopRedactor, which changes nothing. Configure a redactor explicitly. BasicRedactor is a best-effort starting point, not a compliance boundary.

If the application already uses tracing, add the breadcrumb layer:

use tracing_subscriber::prelude::*;

tracing_subscriber::registry()
    .with(tracing_subscriber::fmt::layer())
    .with(faultbox::BreadcrumbLayer::new().only_targets(["myapp"]))
    .init();

Manual breadcrumbs work too, including from libraries. They are inert until the host initializes FaultBox:

faultbox::breadcrumb!(Info, "myapp.store", "starting page read", { "class": "fts" });

At the detection site, emit one capture for the layer that knows the invariant:

use faultbox::{context, Capture, EventKind};

let ctx = context::Adhoc {
    kind: "store.authenticated_page_read",
    key: "aead-read-failure".to_owned(),
    value: faultbox::serde_json::json!({
        "component": "fts",
        "operation": "page-read",
    }),
};

let _report_dir = Capture::new(
    EventKind::Corruption,
    "authenticated page failed validation during read",
)
.error_chain(faultbox::error_chain_of(&err))
.domain(&ctx)
.preserve(
    "store-snapshot",
    store_snapshot_path,
    "store.corrupt",
    Some("snapshot captured at detection site".to_owned()),
)
.emit();

The grouping key is the failure class, not the page, record, tenant, path, or user-specific instance. A stable key like "aead-read-failure" groups the same bug. A key that embeds a page id creates a new group for every occurrence and defeats coalescing.

emit is synchronous. It hashes, copies, fsyncs, and locks local filesystem state. That is deliberate because it must work from places where an async runtime may be unusable, including a panic hook. In an async application, call it from spawn_blocking or the equivalent for your runtime when the call is not already on a blocking path.

To read reports back:

for group in faultbox::reader::list("faultbox-reports")? {
    println!("{}", group.summary());
}

Using FaultBox through a dependency stack

The dependency-stack rules are where it is easiest to get a technically correct crate and a useless integration.

First, only the binary calls faultbox::init. Initialization is process-global and first-init-wins. Later calls return false and change nothing. A library that initializes the recorder is taking ownership of a directory, redactor, and hook policy it should not own.

Second, report once, at the detection site. If PageDB detects an authenticated-page failure, PageDB is the layer with the typed invariant. If NodeDB-Lite only sees a propagated storage error, it should not create a second report for the same root cause unless it is detecting a distinct failure class. Layers above should add breadcrumbs and preserve typed errors.

Third, breadcrumbs merge. If each layer records meaningful breadcrumbs into the same initialized process, a low-level report can still carry high-level context such as the operation class that led to the read.

Fourth, enable downstream diagnostics explicitly. If you want breadcrumbs and captures through a dependency stack, make them an intentional feature path. Do not assume a downstream crate's optional diagnostics are enabled because the top-level application depends on FaultBox.

Finally, make sure there is one Cargo identity for faultbox:

cargo tree -d

If one crate depends on a path checkout and another depends on the published version, Cargo can compile two different faultbox packages. That gives you two independent sets of process-global state. The binary may initialize one while a library emits through the other, and those captures will disappear because that copy was never initialized.

Optional native-crash capture

Native crash capture is optional and host-only. Enable the feature only if you need out-of-process minidumps for native faults. The implementation targets signals such as SIGSEGV and abort as well as stack overflow.

[dependencies]
faultbox = { version = "=0.1.2", features = ["native-crash"] }

The setup is sensitive to order. run_crash_monitor_if_env() must be the first statement in main, before argument parsing, logging setup, config loading, lock acquisition, or any other application work. The monitor is a re-exec of the same binary. This call is what lets the monitor process become the monitor instead of starting the application again.

fn main() {
    if faultbox::run_crash_monitor_if_env() {
        return;
    }

    faultbox::init(
        faultbox::Config::new(
            "myapp",
            env!("CARGO_PKG_VERSION"),
            std::path::PathBuf::from("faultbox-reports"),
        )
            .redactor(Box::new(faultbox::BasicRedactor::new()))
            .install_native_crash_handler(true),
    );

    run_app();
}

fn run_app() {}

Enabling the Cargo feature does not arm the handler by itself. The application has to opt in with install_native_crash_handler(true).

Privacy, security, and operational limits

FaultBox is std-only and writes synchronously to the local filesystem. It has no SaaS backend, no network uploader, no detector, no symbolicator, no async runtime, and no durable report migration system.

The recorder can redact strings in reports, but you have to configure redaction. NoopRedactor is the default. BasicRedactor is best-effort and should be treated as a starting point. If your application handles regulated or high-sensitivity data, write a project-specific redactor and test it with your real data shapes.

Preserved artifacts and minidumps are verbatim sensitive data. They are not redacted. A preserved store is a copy of the store. A minidump can contain process memory. Put the reports directory somewhere private, bound retention, and think about support workflows before asking users to send reports.

On Unix, FaultBox creates report directories and files with owner-only modes. On Windows, it relies on the parent directory ACL. That means the parent location still matters.

The current defaults are bounded, not magic: 128 breadcrumbs, 256 MiB maximum preserved artifact size, and best-effort retention of 64 report groups or 2 GiB of recorded artifacts. Tune them for the application. A desktop app, an embedded database, and a server process do not have the same disk budget.

On wasm32-unknown-unknown, FaultBox is inert because there is no local filesystem recorder. Native crash capture and the shared-memory ring are host-only features.

What it does not do

FaultBox does not find corruption by itself. Your code has to detect the failed checksum, bad authentication tag, impossible state, or invariant violation and emit a capture.

It does not replace regression tests. It should help you get from an unreproducible incident to an understood bug. Once you understand the bug, write the test.

It does not replace logs. Logs are still the right place for routine observability and operator-facing state.

It does not replace panic reporters for teams that only need panic reporting. If panic reports already solve the problem, use the simpler tool.

It does not upload reports, symbolize stack traces for you, run a service, or deduplicate across a fleet. If you parse FaultBox reports with your own tooling, keep the parser tied to the FaultBox version that produced them because the format may evolve before 1.0.

It does not make sensitive artifacts safe to share. Redaction applies to report strings. Preserved files, directories, and minidumps are raw evidence.

Preserve the evidence, then return to testing

The original incident did not become understandable because I guessed harder. It became understandable because the application kept the evidence from the moment it still knew what kind of failure had happened.

That is the useful artifact FaultBox is trying to produce: enough local, bounded, structured evidence to convert "we cannot reproduce this" into "we understand this bug."

After that, the job returns to ordinary engineering. Fix the boundary. Add the regression test. Keep the report as the reason you stopped guessing.

---

Try FaultBox before the next failure disappears

If you are chasing a corruption bug, invariant failure, or crash that refuses to happen under a test, get FaultBox from crates.io. Start with one explicit capture at the point where your code still knows what went wrong, then add breadcrumbs around the operations you would want to reconstruct later.

If you use it, I want to know what it captured, what evidence was still missing, and where the integration felt awkward. You can leave a comment here or open an issue on GitHub.

Have you spent hours chasing a failure that only appeared in a real workload? Would a black-box recorder have changed how you investigated it?