I Built a Crash-Safe Key-Value Store Without sqlite3 — Then I Killed It

arfaaa243 — Fri, 04 Sep 2026 05:09:53 +0000

I built Anvil, a crash-safe, embedded, log-structured key-value store for Hackathon Raptors' Zero Dependency Hackathon 2026 — Track D: Data & Storage.

The constraint was simple:

No third-party runtime dependencies.

But I didn't want to solve that by wrapping sqlite3 and calling it a storage engine.

So I built the storage layer myself:

append-only log
binary record format
in-memory index
CRC32 integrity checks
crash recovery
single-writer locking
atomic compaction
verification tooling
reproducible single-file builds

All with Python's standard library.

The interesting part wasn't making this work:

put(key, value)
get(key)
delete(key)

The interesting part was answering a harder question:

What should a storage engine believe after the process dies halfway through a write?

That's where Anvil really started.

Why build another key-value store?

There are already excellent storage options in Python.

You can use sqlite3.

You can use shelve or dbm.

You can install something like diskcache.

So why build another one?

Because Track D wasn't really asking:

"Can you make a dictionary that persists?"

It was asking what happens when you own the storage layer.

That meant Anvil couldn't simply hide persistence behind another database engine.

I wanted every important guarantee to be visible in the code:

write → persist → recover → verify → compact

No black box.

The first design decision: append, don't overwrite

The core of Anvil is an append-only log.

Instead of finding an old value on disk and modifying it in place, every operation becomes a new record.

For example:

PUT user → Alice
PUT user → Bob
DELETE user
PUT user → Charlie

The log contains the history.

The in-memory index tells Anvil where the latest record for each key lives.

So the architecture becomes:

             CLI / Python API
                    │
                    ▼
               ┌─────────┐
               │  Store  │
               └────┬────┘
                    │
             ┌──────┴──────┐
             ▼             ▼
       In-memory Index   Log File
                           │
                           ▼
                    Binary Records
                           │
                           ▼
                      CRC32 Check
                           │
                           ▼
                         Disk

This shape gives us something important:

the index is not the source of truth.

The persisted log is.

If the process disappears, Anvil can scan the log again and rebuild the index.

That's a much more useful property for a crash-safe storage engine than simply keeping a dictionary in memory.

What does a record actually look like?

I didn't use pickle or an external serialization package.

Anvil uses Python's struct module to define its own binary record format.

Conceptually, a record looks like:

┌──────────┬─────────┬─────────┬────┬─────┬─────┬───────┐
│ checksum │ key_len │ val_len │ op │ seq │ key │ value │
│  4 bytes │ 4 bytes │ 4 bytes │ 1B │ 8B  │ ... │  ...  │
└──────────┴─────────┴─────────┴────┴─────┴─────┴───────┘

The checksum is generated with zlib.crc32.

The sequence number gives records an ordering.

The lengths tell the reader exactly how to split the key and value.

And those length fields turned out to be more important than I initially expected.

The subtle corruption case

A checksum sounds simple:

Calculate a checksum of the data and reject it if the checksum doesn't match.

But what exactly counts as "the data"?

Suppose the checksum only covered:

key + value

Now imagine someone changes:

key_len -= 1
val_len += 1

The total body size can remain exactly the same.

The underlying bytes haven't changed.

So a checksum covering only key + value could still pass.

But the parser would now interpret the same bytes using different boundaries.

That's a nasty failure mode because the record can look structurally valid while being interpreted incorrectly.

The fix was to include the length fields themselves in the checksum input:

key_len
val_len
op
seq
key
value

Anvil has dedicated regression tests for exactly this class of corruption, including a case where the corrupted lengths still add up to the correct total body size.

That's the kind of bug that is easy to miss if the storage format is treated as "just some bytes."

Then I actually killed it

A crash-safe claim is easy to write in a README.

Testing it is harder.

So Anvil has a real crash harness that starts a writer, lets it perform writes, and then kills the process with SIGKILL.

Then a new process reopens the store.

The question isn't:

"Did Python throw an exception?"

The question is:

"After the process was brutally terminated, can the storage engine still recover every acknowledged write without returning garbage?"

The write-path crash harness passed 8/8 real SIGKILL iterations.

Each run killed the process after a different number of acknowledged writes, and the recovered store remained usable with the acknowledged data intact.

That test changed how I think about persistence.

A successful write() call is not the end of the story.

The real question is what survives after the process is gone.

Not every damaged record means the same thing

One important recovery decision in Anvil is distinguishing between a truncated final write and actual corruption.

Imagine the process dies here:

[valid record]
[valid record]
[valid record]
[half-written record]
               ↑
             crash

That's a truncated tail.

The engine can recognize that the final record isn't complete.

But this is different:

[valid record]
[valid record]
[CORRUPTED RECORD]

If a complete record fails its checksum, Anvil doesn't silently guess what happened.

It surfaces corruption.

That's deliberate.

A storage engine should be conservative about what it considers trustworthy.

Compaction: the crash problem doesn't disappear

An append-only log eventually contains obsolete records.

If I write:

PUT counter → 1
PUT counter → 2
PUT counter → 3
PUT counter → 4

only the latest value is needed for normal reads.

So Anvil has compaction.

But compaction introduces another dangerous moment:

What if the process dies while creating the new compacted file?

I didn't want a half-written compacted file to replace the original.

So the process is roughly:

old log
   │
   ▼
write temporary compacted file
   │
   ▼
fsync temporary file
   │
   ▼
atomic os.replace()
   │
   ▼
fsync directory where supported

The original log remains untouched until the replacement is ready.

And this isn't just described in the documentation.

Anvil has a separate compaction crash harness.

The documented test ran 8/8 real SIGKILL iterations successfully, preserving all 4,000 pre-compaction key/value pairs and allowing compaction to be retried afterwards.

Where Python's standard library actually made me suffer

"Zero dependencies" sounds easy if you only look at requirements.txt.

It's much less easy when you start implementing the missing pieces yourself.

Here's what Anvil uses:

Problem	What Anvil uses
Binary record format	`struct`
Integrity checking	`zlib.crc32`
File I/O / durability	`os`, `io`
File locking	`fcntl` / `msvcrt`
CLI	`argparse`
Testing	`unittest`
Benchmarking	`time.perf_counter`
Deterministic test data	`random.Random`
Build artifact	standard-library `zipfile` tooling

The interesting part isn't the module list.

It's what those modules forced me to implement myself.

There was no storage library handling the record layout.

No external locking library handling writers.

No serialization framework deciding how objects should be persisted.

No testing framework giving me the crash semantics.

The standard library gave me primitives.

I had to build the policy around them.

I deliberately didn't use `sqlite3`

This deserves its own section because sqlite3 is already in Python's standard library.

Technically, I could have used it.

But then the project would mostly be:

Anvil
  ↓
sqlite3
  ↓
disk

That would defeat the interesting part of Track D.

Anvil instead owns:

the binary format
the log
the index
recovery
checksums
locking
compaction

The point wasn't to prove that SQLite is bad.

It's the opposite.

SQLite is extremely mature.

The point was to understand what sits underneath a persistent key-value abstraction when you don't delegate that layer.

The number I wasn't going to hide

Here's the uncomfortable part.

Anvil's benchmark isn't trying to win a raw throughput contest.

With:

5,000 writes
5,000 reads
100-byte values

the benchmark reported:

Metric	Result
Write time	0.7247 s
Write throughput	6,899.5 ops/sec
Read time	0.0250 s
Read throughput	200,015.8 ops/sec
Database size	695,000 bytes

These are measurements from the benchmark shipped with the repository, not a theoretical estimate.

And yes:

the writes are much slower than a memory-speed append.

That's intentional.

Anvil calls fsync after every write because durability is part of the design.

So the write path is paying for the storage guarantee:

append
  ↓
flush
  ↓
fsync
  ↓
acknowledge

The result is that write performance is bounded heavily by storage latency.

I could make the benchmark number look much better by weakening the durability policy.

I chose not to.

For this project, a slower honest write is more useful than a fast write with an unclear durability guarantee.

So is Anvil faster than diskcache?

No.

And I'm not going to pretend it is.

diskcache is a mature project with a much broader feature set.

Anvil is targeting a narrower problem:

A small persistent key-value storage layer where the application wants to own the storage implementation and keep the runtime dependency surface at zero.

Anvil gives you:

put
get
delete
scan
persistence
crash recovery
corruption detection
compaction
single-writer protection

It does not try to reproduce every feature of a mature cache library.

No TTL.

No eviction policy.

No memoization framework.

No distributed replication.

No multi-key transactions.

That's an important distinction.

Anvil is a focused storage engine, not a replacement for every database or cache.

Making the claims easy to verify

One thing I wanted to avoid was a README full of claims that a judge simply had to trust.

So Anvil has a verification mindset built into the project.

For example:

python scripts/check_deps.py

checks the actual imports used by the source.

The repository documents the expected result as standard-library-only imports with an empty dependency manifest.

There is also a dedicated verify command that can inspect the storage file and report:

path: user.anvil
file size: 46 byte(s)
valid record count: 2
put records: 2
delete records: 0
live key count: 2
verified up to offset: 46
result: PASS

So verification isn't just:

"Trust me, the database is fine."

It's a separate operation.

Testing more than the happy path

The test suite doesn't only check:

put → get → success

It covers things such as:

persistence across restart
delete semantics
empty values
large values
malformed records
checksum corruption
corrupted length fields
truncated final records
concurrent writer handling
lock release
compaction correctness
crash recovery during compaction
CLI exit codes
reproducible-build verification

The current project documentation records a 100-test unittest suite covering these behaviors.

The important part isn't the number 100 by itself.

It's that the tests are aimed at the places where storage engines usually become dangerous:

partial writes, corruption, concurrency and replacement.

I also wanted the build to be reproducible

The final artifact can be built deterministically.

The build script fixes things such as:

file ordering
timestamps
platform flags
permissions
compression settings

Two builds from the same source produced the same SHA-256:

fc5b367929a65c9db9e859eda91b491f8c59ef5d1b79e6ccaaf84ab062c7ffd4

for both artifacts.

There's also a deliberately stated limitation:

this proves byte-identical output for the tested build environment/toolchain; it isn't a claim that arbitrary future Python or zlib versions will necessarily produce the same compressed bytes.

Again, the goal is not to make the claim sound bigger.

It's to make the claim precise.

What I learned

When I started, the interesting problem looked like:

"How do I implement a key-value store?"

By the end, that wasn't really the problem.

The interesting questions were:

When is a write durable?

What does a crash leave behind?

Which bytes can I trust?

What happens if a length field is corrupted but the record still has the right total size?

What happens if compaction dies halfway through?

What happens if another process tries to write at the same time?

Those questions shaped almost every important part of Anvil.

And that's probably the biggest lesson I got from building it:

Writing data to disk is easy. Knowing when you can trust that data is the real problem.

What Anvil doesn't try to be

Anvil is intentionally small.

It is:

embedded
persistent
log-structured
single-writer
crash-recoverable
standard-library-only

It is not:

a distributed database
a full SQL database
a multi-writer transactional engine
a feature-complete cache
a cryptographically authenticated storage system

CRC32 gives integrity detection.

It does not provide cryptographic authenticity.

Those boundaries are intentional.

A project becomes more useful when its guarantees are clear.

Final Thoughts

I started Anvil because the Zero Dependency constraint made me ask a slightly uncomfortable question:

What if I couldn't rent the storage layer from someone else?

The answer turned out to be much more than a dictionary backed by a file.

It meant designing a record format.

It meant deciding what a checksum actually protects.

It meant rebuilding the index after a crash.

It meant distinguishing a truncated write from real corruption.

It meant making compaction atomic.

It meant testing the system by actually killing it.

And it meant accepting a slower write path because durability mattered more than a pretty benchmark number.

Anvil is still a small storage engine.

But building it changed the way I think about persistence.

The hard part isn't storing the value.

The hard part is knowing when you can trust it.

Built for Hackathon Raptors' Zero Dependency Hackathon 2026 — Track D: Data & Storage.

Repository: Anvil on GitHub
Repository: https://github.com/arfaaa243/anvil-db

Zero runtime dependencies.

Real crash testing.

Own storage format.

No sqlite3 underneath.

DEV Community: arfaaa243