DEV Community: codewitharyan29

Killed My Own Process 300 Times to Prove ChronoVault Doesn't Lie About Durability

codewitharyan29 — Thu, 03 Sep 2026 15:27:27 +0000

TL;DR: I built ChronoVault v2 for Hackathon Raptors' Zero Dependency Hackathon (Track D — Data & Storage) — a content-addressable snapshot and recovery engine, zero runtime dependencies, backed by a hand-rolled pack-file format. A comment saying "this is atomic" is worth nothing to a judge, so instead of asserting crash-safety, I hard-killed my own writer process mid-write, over and over, and made the vault prove it survived. This is that story, plus a benchmark number I didn't want to publish and three Windows bugs that only showed up on real hardware.

279 tests. 20 CLI commands. 0 runtime dependencies. judge_mode.py VERIFIED on Python 3.11 and 3.14.

Proving crash-safety instead of asserting it

Anyone can write "atomic writes, crash-safe" in a README. It costs nothing and proves nothing. So recover-check is tested against a real SIGKILL, not a simulated one: a writer subprocess runs a snapshot, gets hard-killed at a randomized point mid-write, and the vault is reopened cold — no manual repair step allowed.

The test checks three things, not one:

Every snapshot acknowledged before the kill is intact.
The in-flight snapshot at the moment of the kill is either cleanly absent or a discarded partial write — never garbage returned from a read.
The vault is immediately usable afterward, with no repair command required.

This is also where the pack-quarantine logic earned its place instead of being a "nice to have." Early in testing, a kill landing mid-pack-write left a truncated pack that the recovery scan initially tried to read as valid — and returned a wrong length from. Quarantine — isolate anything that fails its checksum before it ever reaches the index — closed that hole. The fix wasn't "trust the file less," it was "never let an unverified pack answer a read at all."

Run it enough times and the interesting failures stop being about the crash itself and start being about the code paths a crash exposes that a clean shutdown never does. That's the actual value of hard-killing your own process 300 times instead of once: the first ten runs prove the happy path works, and the next 290 are what find the pack-quarantine gap.

Architecture at a glance

flowchart TD
    subgraph L1["CLI"]
        CLI["cli.py — 20 subcommands"]
    end

    subgraph L2["Core Engine"]
        Snap["snapshot.py<br/>atomic rename + fsync"]
        CAS["Content-Addressed Store<br/>hashlib SHA-256, pack files"]
        Idx["path_history.py<br/>rename-aware lineage"]
        Rec["recover.py<br/>recover-check, SIGKILL-tested replay"]
    end

    subgraph L3["Persistence"]
        Pack[("Pack files on disk")]
        Quar["Pack quarantine<br/>corrupt/truncated packs isolated"]
    end

    CLI --> Snap
    CLI --> CAS
    CLI --> Idx
    CLI --> Rec
    Snap --> CAS
    Snap --> Idx
    CAS --> Pack
    Idx --> Pack
    Rec -.->|verified via SIGKILL injection| Snap
    Rec --> Pack
    Pack --> Quar

    style L3 fill:#2d2d2d,stroke:#f5a623,stroke-width:2px,color:#fff
    style Rec fill:#2d2d2d,stroke:#f5a623,stroke-width:2px,color:#fff

The shape that matters is where recovery sits: it reads pack files directly, never through the CLI or the snapshot writer, the same way a verify-style tool deliberately bypasses the normal open path. That's deliberate — if the write path itself is what crashed, recovery can't depend on any code the crash might have left half-updated.

The number I didn't want to publish

Every other number in this post makes ChronoVault look good. This one doesn't.

I benchmarked vault snapshot against diskcache — the closest real-world comparison for local content-addressed storage:

Metric	ChronoVault	diskcache
Write throughput	baseline	9–13x faster
On-disk size, repeated / near-duplicate snapshots	5–47x smaller	baseline

Why it loses on writes: every vault snapshot call pays for a full content hash of each file plus a pack-index lookup before a single byte is committed, because deduplication has to happen before the write, not after. diskcache skips that entirely. That's the whole trade in one sentence: I pay a hashing-and-lookup tax on every write so a hundred near-identical snapshots don't cost a hundred times the disk.

If your workload is write-latency-bound, diskcache is the right tool and I'm not pretending otherwise. ChronoVault wins when the workload is storage-bound — long retention windows, many near-duplicate snapshots, disk-constrained environments. A real loser column next to a real winner column, not a benchmark that only shows its best number.

Where the standard library actually made me suffer

Three places, all invisible until I tested on real Windows hardware instead of assuming POSIX behavior generalizes.

multiprocessing's spawn context raised a KeyboardInterrupt that wasn't one. Worker processes died mid-operation with nobody near Ctrl+C. Windows' spawn-based multiprocessing ties process teardown to _winapi.WaitForSingleObject, which can surface as a spurious interrupt with no signal ever sent — a path fork-based Unix multiprocessing never touches. I misdiagnosed this three separate times as my own bug before proving otherwise, using a detached Start-Job test that ran the worker fully isolated from the parent's console.

WinError 32 — Windows refusing to delete a file it considers still open. POSIX lets you unlink() an open file; the inode survives until the last handle closes. Windows refuses the delete outright. Handling it meant retrying past the handle-release window instead of importing psutil to hide it.

A join-before-drain ordering bug deadlocked the IPC queue. worker.join() was called before the result queue was fully drained. If the OS-level pipe buffer filled before the parent drained it, the child blocked on a full pipe and the parent blocked on join() — a deadlock that only appears under load. Fixed by draining before joining.

None of these are exotic. They're the specific price of touching multiprocessing and cross-process queues without a battle-tested wrapper library between you and the OS.

The edge case that ate an afternoon

Rename-aware file lineage in the path-history index. The question sounds simple — "was this file renamed, or deleted and recreated?" — until content, path, and inode can each change independently, sometimes in the same tick, with zero heuristics borrowed from someone else's diffing library to lean on.

The ambiguous case that actually happened: a file renamed and edited in the same snapshot cycle. "Renamed A→B, then edited" and "A deleted, unrelated B created" are both defensible reads of the same delta, and guessing wrong either invents a false history or silently drops a real one.

I didn't solve it by getting cleverer. I wrote down, explicitly, which cases the lineage tracker resolves correctly, and which ones it honestly doesn't try to disambiguate. A documented limitation beats a confident guess that's wrong on the exact case a judge tests.

What I'd do differently

The Windows CI matrix went in after most of the Windows-specific bugs had already been found manually, one laptop test at a time. Every area I later ran through real Windows hardware for the first time produced at least one genuine finding. If Ubuntu + Windows CI had existed from the start, at least two of those bugs would have surfaced in an automated run instead of on my own machine, days closer to the deadline than I'd have liked.

Where it landed

279 tests passing on Python 3.11 and 3.14, judge_mode.py VERIFIED
20 CLI commands, zero third-party runtime dependencies
Crash-safety proven, not asserted — recover-check verified against real SIGKILL injection, with pack quarantine closing the gap testing found
CI green across Python 3.10–3.14 × Ubuntu + Windows
STDLIB.md documents 14 real substitutions, each cross-referenced to the file that uses it
All four bonus categories claimed (+16) — Single File, Reproducible Build, Package Killer re-earned via the diskcache numbers above, STDLIB Log

python judge_mode.py

runs the full verification — test suite, single-file hash check, isolated-mode dependency audit, and the content-addressing proof's 8 invariants — as one command, so a judge doesn't have to trust this post; they can re-derive it.

Closing thought

Anyone with an AI coding agent can produce a CLI that hashes files and calls it content-addressed storage. The hard part was building enough ways to catch myself being wrong — a crash-safety claim verified with a real SIGKILL instead of a comment asserting atomicity, a benchmark table honest enough to put a 9–13x loss next to a 5–47x win, and a judge_mode.py that re-derives every claim in this post instead of asking a judge to trust it. The vault is the artifact. The willingness to kill it 300 times and publish what survived is the actual submission.

Repo: github.com/codewitharyan29/ChronoVault-v2
Track: D — Data & Storage

Built for Zero Dependency 2026, run by @partnerships_raptors.

ZeroDependencyHack #Python #OpenSource #DeveloperTools #CLI

PythonStandardLibrary #BuildInPublic

Porting natsort to Rust taught me that "it compiles" is the easy 5%

codewitharyan29 — Thu, 06 Aug 2026 14:44:40 +0000

TL;DR: I ported Python's natsort library to Rust for Port Mortem 2026.

Writing the sorting algorithm turned out to be the easy part.

The difficult part was proving the Rust implementation behaved identically to the original Python library across thousands of edge cases without quietly changing behavior.

That meant treating verification as the real project.

Why I chose natsort

Most demo ports are calculators, parsers, or utilities with a handful of tests.

I wanted something that would actually stress the verification process.

natsort sorts strings the way humans expect:

file2
file10
file20

instead of lexicographic order.

That sounds simple until you discover it supports:

seven algorithm variants
signed and floating-point numbers
scientific notation
full Unicode numeric characters
locale-aware behavior
Roman numerals
fractions
a real upstream test suite covering years of edge cases

A port that looks correct on five examples proves almost nothing.

A port that survives the library's own test suite, differential fuzzing, CLI comparisons, property testing, and mutation testing starts becoming evidence.

The architecture

The verification pipeline ended up looking like this:

             Python Test Suite
                    │
                    ▼
          Thin Python Adapter
                    │
                    ▼
             Compiled Rust Binary
                    │
                    ▼
          Rust natsort Implementation
                    │
                    ▼
 Compare:
 • stdout
 • stderr
 • exit codes
 • ordering
 • fuzz results
 • properties

The adapter intentionally contains almost no logic.

Its only job is forwarding Python test inputs to the Rust implementation.

That design choice turned out to matter more than I expected.

What broke (and why those bugs mattered)

1. Differential fuzzing found a bug in the original library

Differential fuzzing generates thousands of random inputs and compares both implementations.

Most mismatches were my mistakes.

One wasn't.

>>> natsorted(["1e400", "1e500"], alg=ns.REAL)
['1e400', '1e500']

>>> natsorted(["1e500", "1e400"], alg=ns.REAL)
['1e500', '1e400']

Both calls succeed.

Both disagree.

The reason is surprisingly simple.

Both values overflow to floating-point infinity.

Since both become inf, the comparison reports them equal, and Python's stable sort simply preserves whichever order the inputs arrived in.

Rust's f64 behaves the same way.

That means the correct port is not one that "fixes" the behavior.

The correct port reproduces it.

I reported the issue upstream:

natsort#192

Finding a real bug in a mature library simply by holding another implementation beside it was one of the most satisfying moments of the project.

2. My adapter was lying to me

I assumed the adapter was a thin wrapper.

It wasn't.

One function calculated the correct Rust-backed answer...

...and then ignored it.

Instead it silently fell back to a handwritten Python implementation that didn't understand signed numbers or Unicode.

index_natsorted(["-5", "-1", "3"], alg=ns.REAL)

returned:
[2, 1, 0]

correct:
[0, 1, 2]

Differential testing doesn't automatically catch bugs in the adapter.

You have to verify the adapter itself.

That lesson probably saved me more time than any optimization.

3. Windows disagreed with Linux

Everything looked perfect.

Until I ran it on Windows.

Five bugs immediately appeared.

executable needed .exe
python3 didn't exist (py does)
subprocess output defaulted to CP-1252 instead of UTF-8
file reads had the same encoding problem
Unix-only resource crashed benchmarks entirely

None of those appeared on Linux.

None appeared in my development environment.

Every one appeared on a real Windows machine.

"Works on my machine" turned out to be the weakest verification strategy of all.

4. GitHub Actions lied with a green checkmark

The deployment workflow looked successful.

The live website returned 404.

The culprit?

A multi-line shell block:

curl ... | sh
build
deploy

The installation silently failed.

The following commands still exited successfully.

GitHub reported the entire step as green.

The fix wasn't clever.

It was boring.

I switched to:

cargo install wasm-pack

and added a deployment check that refuses to publish unless the expected build artifacts actually exist.

A green checkmark only matters if it's checking the thing you care about.

How I proved behavioral equivalence

The implementation wasn't the deliverable.

Evidence was.

I ended up with six independent layers of verification.

1. Upstream tests

Run the original natsort test suite without modifying it.

Tests excluded only when Rust genuinely cannot reproduce Python-specific behavior, and every exclusion is documented.

2. Differential fuzzing

27,000 seeded comparisons across all algorithm variants.

Same inputs.

Same outputs.

Reproducible failures.

3. CLI differential testing

2,800 CLI invocations comparing:

stdout
stderr
exit codes

Matching output alone isn't enough.

Programs communicate failure through exit codes too.

4. Property testing

Randomized verification of:

idempotence
antisymmetry
transitivity

These find classes of bugs that handwritten examples never will.

5. Mutation testing

I deliberately broke my implementation.

reversed comparisons
removed signs
altered parsing logic

If the verification pipeline couldn't detect injected bugs, it wasn't trustworthy.

Every injected mutation was caught.

6. Honest benchmarking

Instead of publishing a single "10× faster" headline, I reported:

mean
median
p95
p99
peak memory

The real result was 6–8× faster, depending on workload.

Honest numbers build more confidence than inflated ones.

The decision I'd take back

I spent a huge amount of time chasing perfect Unicode parity.

It was technically rewarding.

It pushed differential matching from roughly 43% to 100% across the full Unicode character set.

I'm proud of that work.

But I crossed the point of diminishing returns.

One adapter-level discrepancy survived until the deadline on a single CI runner.

I excluded that specific test rather than claim support I couldn't fully defend.

Looking back, I'd stop polishing Unicode earlier and spend those hours investigating the remaining verification discrepancy instead.

Verification gets exponentially more expensive near the finish line.

Knowing when to stop is part of engineering.

Final numbers

By the end of the project:

Original upstream tests running against Rust
27,000 differential fuzz comparisons
2,800 CLI differential runs
Property testing
Mutation testing
Honest benchmark suite
6–8× speedup
One upstream bug reported
Cross-platform verification on Linux and Windows

Reproduce everything

Everything needed to verify the project is public.

Demo

https://youtu.be/jVz_2AP4U4w

Live WASM demo + verification dashboard

https://codewitharyan29.github.io/Port-Mortem/

Repository

https://github.com/codewitharyan29/Port-Mortem

Upstream issue

https://github.com/SethMMorton/natsort/issues/192

**Hackathon rapters : https://www.raptors.dev/project/code-resurrection-2026-port-mortem

Closing thoughts

Anyone with an AI coding agent can produce something that compiles.

Compilation is the easy part.

The difficult part is proving—without hand-waving—that another implementation behaves the same as the original across thousands of edge cases, different operating systems, different execution paths, and even the original project's own bugs.

By the end of this project, I stopped measuring success by whether the Rust code compiled.

I measured success by how much evidence I had that it behaved like Python.

Built for Port Mortem / Code Resurrection by Hackathon Raptors

#PortMortem2026 #HackathonRaptors #Rust #Python #OpenSource #SystemsProgramming #Testing #Verification #Fuzzing
**