I built Anvil, a crash-safe, embedded, log-structured key-value store for Hackathon Raptors' Zero Dependency Hackathon 2026 — Track D: Data & Storage.
The constraint was simple:
No third-party runtime dependencies.
But I didn't want to solve that by wrapping sqlite3 and calling it a storage engine.
So I built the storage layer myself:
- append-only log
- binary record format
- in-memory index
- CRC32 integrity checks
- crash recovery
- single-writer locking
- atomic compaction
- verification tooling
- reproducible single-file builds
All with Python's standard library.
The interesting part wasn't making this work:
put(key, value)
get(key)
delete(key)
The interesting part was answering a harder question:
What should a storage engine believe after the process dies halfway through a write?
That's where Anvil really started.
Why build another key-value store?
There are already excellent storage options in Python.
You can use sqlite3.
You can use shelve or dbm.
You can install something like diskcache.
So why build another one?
Because Track D wasn't really asking:
"Can you make a dictionary that persists?"
It was asking what happens when you own the storage layer.
That meant Anvil couldn't simply hide persistence behind another database engine.
I wanted every important guarantee to be visible in the code:
write → persist → recover → verify → compact
No black box.
The first design decision: append, don't overwrite
The core of Anvil is an append-only log.
Instead of finding an old value on disk and modifying it in place, every operation becomes a new record.
For example:
PUT user → Alice
PUT user → Bob
DELETE user
PUT user → Charlie
The log contains the history.
The in-memory index tells Anvil where the latest record for each key lives.
So the architecture becomes:
CLI / Python API
│
▼
┌─────────┐
│ Store │
└────┬────┘
│
┌──────┴──────┐
▼ ▼
In-memory Index Log File
│
▼
Binary Records
│
▼
CRC32 Check
│
▼
Disk
This shape gives us something important:
the index is not the source of truth.
The persisted log is.
If the process disappears, Anvil can scan the log again and rebuild the index.
That's a much more useful property for a crash-safe storage engine than simply keeping a dictionary in memory.
What does a record actually look like?
I didn't use pickle or an external serialization package.
Anvil uses Python's struct module to define its own binary record format.
Conceptually, a record looks like:
┌──────────┬─────────┬─────────┬────┬─────┬─────┬───────┐
│ checksum │ key_len │ val_len │ op │ seq │ key │ value │
│ 4 bytes │ 4 bytes │ 4 bytes │ 1B │ 8B │ ... │ ... │
└──────────┴─────────┴─────────┴────┴─────┴─────┴───────┘
The checksum is generated with zlib.crc32.
The sequence number gives records an ordering.
The lengths tell the reader exactly how to split the key and value.
And those length fields turned out to be more important than I initially expected.
The subtle corruption case
A checksum sounds simple:
Calculate a checksum of the data and reject it if the checksum doesn't match.
But what exactly counts as "the data"?
Suppose the checksum only covered:
key + value
Now imagine someone changes:
key_len -= 1
val_len += 1
The total body size can remain exactly the same.
The underlying bytes haven't changed.
So a checksum covering only key + value could still pass.
But the parser would now interpret the same bytes using different boundaries.
That's a nasty failure mode because the record can look structurally valid while being interpreted incorrectly.
The fix was to include the length fields themselves in the checksum input:
key_len
val_len
op
seq
key
value
Anvil has dedicated regression tests for exactly this class of corruption, including a case where the corrupted lengths still add up to the correct total body size.
That's the kind of bug that is easy to miss if the storage format is treated as "just some bytes."
Then I actually killed it
A crash-safe claim is easy to write in a README.
Testing it is harder.
So Anvil has a real crash harness that starts a writer, lets it perform writes, and then kills the process with SIGKILL.
Then a new process reopens the store.
The question isn't:
"Did Python throw an exception?"
The question is:
"After the process was brutally terminated, can the storage engine still recover every acknowledged write without returning garbage?"
The write-path crash harness passed 8/8 real SIGKILL iterations.
Each run killed the process after a different number of acknowledged writes, and the recovered store remained usable with the acknowledged data intact.
That test changed how I think about persistence.
A successful write() call is not the end of the story.
The real question is what survives after the process is gone.
Not every damaged record means the same thing
One important recovery decision in Anvil is distinguishing between a truncated final write and actual corruption.
Imagine the process dies here:
[valid record]
[valid record]
[valid record]
[half-written record]
↑
crash
That's a truncated tail.
The engine can recognize that the final record isn't complete.
But this is different:
[valid record]
[valid record]
[CORRUPTED RECORD]
If a complete record fails its checksum, Anvil doesn't silently guess what happened.
It surfaces corruption.
That's deliberate.
A storage engine should be conservative about what it considers trustworthy.
Compaction: the crash problem doesn't disappear
An append-only log eventually contains obsolete records.
If I write:
PUT counter → 1
PUT counter → 2
PUT counter → 3
PUT counter → 4
only the latest value is needed for normal reads.
So Anvil has compaction.
But compaction introduces another dangerous moment:
What if the process dies while creating the new compacted file?
I didn't want a half-written compacted file to replace the original.
So the process is roughly:
old log
│
▼
write temporary compacted file
│
▼
fsync temporary file
│
▼
atomic os.replace()
│
▼
fsync directory where supported
The original log remains untouched until the replacement is ready.
And this isn't just described in the documentation.
Anvil has a separate compaction crash harness.
The documented test ran 8/8 real SIGKILL iterations successfully, preserving all 4,000 pre-compaction key/value pairs and allowing compaction to be retried afterwards.
Where Python's standard library actually made me suffer
"Zero dependencies" sounds easy if you only look at requirements.txt.
It's much less easy when you start implementing the missing pieces yourself.
Here's what Anvil uses:
| Problem | What Anvil uses |
|---|---|
| Binary record format | struct |
| Integrity checking | zlib.crc32 |
| File I/O / durability |
os, io
|
| File locking |
fcntl / msvcrt
|
| CLI | argparse |
| Testing | unittest |
| Benchmarking | time.perf_counter |
| Deterministic test data | random.Random |
| Build artifact | standard-library zipfile tooling |
The interesting part isn't the module list.
It's what those modules forced me to implement myself.
There was no storage library handling the record layout.
No external locking library handling writers.
No serialization framework deciding how objects should be persisted.
No testing framework giving me the crash semantics.
The standard library gave me primitives.
I had to build the policy around them.
I deliberately didn't use sqlite3
This deserves its own section because sqlite3 is already in Python's standard library.
Technically, I could have used it.
But then the project would mostly be:
Anvil
↓
sqlite3
↓
disk
That would defeat the interesting part of Track D.
Anvil instead owns:
- the binary format
- the log
- the index
- recovery
- checksums
- locking
- compaction
The point wasn't to prove that SQLite is bad.
It's the opposite.
SQLite is extremely mature.
The point was to understand what sits underneath a persistent key-value abstraction when you don't delegate that layer.
The number I wasn't going to hide
Here's the uncomfortable part.
Anvil's benchmark isn't trying to win a raw throughput contest.
With:
- 5,000 writes
- 5,000 reads
- 100-byte values
the benchmark reported:
| Metric | Result |
|---|---|
| Write time | 0.7247 s |
| Write throughput | 6,899.5 ops/sec |
| Read time | 0.0250 s |
| Read throughput | 200,015.8 ops/sec |
| Database size | 695,000 bytes |
These are measurements from the benchmark shipped with the repository, not a theoretical estimate.
And yes:
the writes are much slower than a memory-speed append.
That's intentional.
Anvil calls fsync after every write because durability is part of the design.
So the write path is paying for the storage guarantee:
append
↓
flush
↓
fsync
↓
acknowledge
The result is that write performance is bounded heavily by storage latency.
I could make the benchmark number look much better by weakening the durability policy.
I chose not to.
For this project, a slower honest write is more useful than a fast write with an unclear durability guarantee.
So is Anvil faster than diskcache?
No.
And I'm not going to pretend it is.
diskcache is a mature project with a much broader feature set.
Anvil is targeting a narrower problem:
A small persistent key-value storage layer where the application wants to own the storage implementation and keep the runtime dependency surface at zero.
Anvil gives you:
putgetdeletescan- persistence
- crash recovery
- corruption detection
- compaction
- single-writer protection
It does not try to reproduce every feature of a mature cache library.
No TTL.
No eviction policy.
No memoization framework.
No distributed replication.
No multi-key transactions.
That's an important distinction.
Anvil is a focused storage engine, not a replacement for every database or cache.
Making the claims easy to verify
One thing I wanted to avoid was a README full of claims that a judge simply had to trust.
So Anvil has a verification mindset built into the project.
For example:
python scripts/check_deps.py
checks the actual imports used by the source.
The repository documents the expected result as standard-library-only imports with an empty dependency manifest.
There is also a dedicated verify command that can inspect the storage file and report:
path: user.anvil
file size: 46 byte(s)
valid record count: 2
put records: 2
delete records: 0
live key count: 2
verified up to offset: 46
result: PASS
So verification isn't just:
"Trust me, the database is fine."
It's a separate operation.
Testing more than the happy path
The test suite doesn't only check:
put → get → success
It covers things such as:
- persistence across restart
- delete semantics
- empty values
- large values
- malformed records
- checksum corruption
- corrupted length fields
- truncated final records
- concurrent writer handling
- lock release
- compaction correctness
- crash recovery during compaction
- CLI exit codes
- reproducible-build verification
The current project documentation records a 100-test unittest suite covering these behaviors.
The important part isn't the number 100 by itself.
It's that the tests are aimed at the places where storage engines usually become dangerous:
partial writes, corruption, concurrency and replacement.
I also wanted the build to be reproducible
The final artifact can be built deterministically.
The build script fixes things such as:
- file ordering
- timestamps
- platform flags
- permissions
- compression settings
Two builds from the same source produced the same SHA-256:
fc5b367929a65c9db9e859eda91b491f8c59ef5d1b79e6ccaaf84ab062c7ffd4
for both artifacts.
There's also a deliberately stated limitation:
this proves byte-identical output for the tested build environment/toolchain; it isn't a claim that arbitrary future Python or zlib versions will necessarily produce the same compressed bytes.
Again, the goal is not to make the claim sound bigger.
It's to make the claim precise.
What I learned
When I started, the interesting problem looked like:
"How do I implement a key-value store?"
By the end, that wasn't really the problem.
The interesting questions were:
When is a write durable?
What does a crash leave behind?
Which bytes can I trust?
What happens if a length field is corrupted but the record still has the right total size?
What happens if compaction dies halfway through?
What happens if another process tries to write at the same time?
Those questions shaped almost every important part of Anvil.
And that's probably the biggest lesson I got from building it:
Writing data to disk is easy. Knowing when you can trust that data is the real problem.
What Anvil doesn't try to be
Anvil is intentionally small.
It is:
- embedded
- persistent
- log-structured
- single-writer
- crash-recoverable
- standard-library-only
It is not:
- a distributed database
- a full SQL database
- a multi-writer transactional engine
- a feature-complete cache
- a cryptographically authenticated storage system
CRC32 gives integrity detection.
It does not provide cryptographic authenticity.
Those boundaries are intentional.
A project becomes more useful when its guarantees are clear.
Final Thoughts
I started Anvil because the Zero Dependency constraint made me ask a slightly uncomfortable question:
What if I couldn't rent the storage layer from someone else?
The answer turned out to be much more than a dictionary backed by a file.
It meant designing a record format.
It meant deciding what a checksum actually protects.
It meant rebuilding the index after a crash.
It meant distinguishing a truncated write from real corruption.
It meant making compaction atomic.
It meant testing the system by actually killing it.
And it meant accepting a slower write path because durability mattered more than a pretty benchmark number.
Anvil is still a small storage engine.
But building it changed the way I think about persistence.
The hard part isn't storing the value.
The hard part is knowing when you can trust it.
Built for Hackathon Raptors' Zero Dependency Hackathon 2026 — Track D: Data & Storage.
Repository: Anvil on GitHub
Repository: https://github.com/arfaaa243/anvil-db
Zero runtime dependencies.
Real crash testing.
Own storage format.
No sqlite3 underneath.
Top comments (0)