VaultLog: Building a Persistent Key-Value Store with Zero Dependencies

What if you had to build a useful storage engine without installing a single third-party package?

That was the challenge behind VaultLog, my submission for the Zero Dependency 72-Hour Hackathon by Hackathon Raptors.

The rule was simple:

Use only the standard library. No third-party runtime dependencies.

At first, this sounds like a restriction on what you can import.

In practice, it changes how you think about software.

Instead of asking "Which package should I use?", you start asking:

"What is actually happening underneath that package?"

That question became the foundation of VaultLog.

The Idea

VaultLog is a persistent embedded key-value store written entirely in Go 1.27.

It supports the fundamental operations you would expect from a small key-value database:

• Set a value
• Get a value
• Delete a value
• List stored keys
• Inspect storage statistics

The important part is that VaultLog doesn't wrap an existing database engine.

There is no SQLite driver.

There is no BadgerDB.

There is no BoltDB.

There is no external storage library.

The storage layer itself is implemented in the project.

Why Build a Storage Engine?

For most applications, using an existing database is obviously the right decision.

If I were building a normal production application, I wouldn't recommend writing a database from scratch just because I could.

But the Zero Dependency Hackathon created a different engineering question:

How much of a practical storage system can be built using only the language's standard library?

That made a key-value store a particularly interesting problem.

A database package normally hides a lot of complexity:

• How records are persisted
• How data is indexed
• How writes are represented
• How deleted records are handled
• How data is recovered after restart
• How corrupted data is detected
• How concurrent access is synchronized

VaultLog makes those layers explicit.

The Architecture

The core architecture is intentionally simple:

                 VaultLog
                    │
          ┌─────────┴─────────┐
          │                   │
   In-Memory Index       Append-Only Log
          │                   │
          │                   ▼
          │             Persistent File
          │
          ▼
     Fast Lookups

The system has two important components:

1. Append-Only Log

Every write is represented as a record and appended to the storage file.

Instead of modifying existing records in place, VaultLog keeps adding new records.

For example:

SET user:name Tanya
SET user:role Developer
DELETE user:role

The log becomes the historical sequence of operations.

This approach makes persistence straightforward and gives us a natural source from which the database can reconstruct its state.

2. In-Memory Index

Scanning the entire file every time someone asks for a value would be inefficient.

So VaultLog maintains an in-memory index.

Conceptually:

Key                Record Location
-----------------------------------
user:name          → offset 0x120
user:role          → deleted

The index allows VaultLog to locate the latest record associated with a key without scanning the entire log.

This provides average O(1) hash-map lookup for key access, while persistence remains on disk.

Writing Data

A Set operation follows a simple process:

  Set(key, value)
      │
      ▼
  Encode Record
      │
      ▼
Append Record to Log
      │
      ▼
Update In-Memory Index

The important design decision is that the persistent log is updated before the in-memory state becomes authoritative.

This gives the log a durable representation of the operation.

Deleting Data

Deletes are interesting in an append-only system.

We don't physically remove the old record from the file.

Instead, VaultLog writes a delete record.

SET username → Tanya
DELETE username

During recovery, the replay engine sees both operations and knows that the latest operation means the key no longer exists.

This is one of the trade-offs of append-only storage:

Deletes are cheap, but the log grows over time.

A future version could introduce log compaction to remove obsolete records.

Restart Recovery

Persistence isn't particularly useful if the database forgets its state after restarting.

VaultLog therefore reconstructs its in-memory index from the storage log.

The recovery process looks like this:

  Persistent Log
      │
      ▼
Read Records Sequentially
      │
      ▼
   Validate Record
      │
      ▼
  Replay Operation
      │
      ▼
  Rebuild Index

Suppose the log contains:

SET name Tanya
SET role Developer
DELETE role
SET city Meerut

After replaying these records, the reconstructed state becomes:

name → Tanya
city → Meerut

The deleted key doesn't return because its latest operation was a deletion.

This means the in-memory index doesn't need to be persisted separately.

The log is enough to rebuild it.

Data Integrity

Persistent storage introduces another problem:

How do we know that a record hasn't been corrupted?

VaultLog uses the Go standard library's:

hash/crc32

A checksum is stored alongside record data.

During recovery, the checksum can be calculated again and compared with the stored value.

Conceptually:

Record
  │
  ├── Header
  ├── Key
  ├── Value
  └── Checksum

If the calculated checksum doesn't match the stored checksum, the record cannot be trusted.

This gives the storage layer a basic integrity mechanism without importing an external package.

Concurrency

Another feature normally handled by database libraries is concurrent access.

VaultLog uses Go's standard synchronization primitives from:

sync

The storage engine protects shared state so that multiple operations don't modify the in-memory index unsafely.

Again, no external concurrency framework was necessary.

Go already provides the building blocks.

The Packages I Would Normally Use

This is where the challenge became interesting.

In a normal project, I might reach for external packages for several pieces of functionality.

Instead, VaultLog uses standard-library alternatives.

Normally Used	VaultLog Uses
Database package	Custom append-only storage engine
testify	testing
logrus / zap	log / log/slog
External binary encoding library	encoding/binary
External hashing utility	hash/crc32
External filesystem utilities	os / io
External path utilities	path/filepath
External CLI parser	flag

The biggest replacement wasn't a small utility.

It was the database layer itself.

Instead of importing a storage engine, VaultLog implements one.

The Hardest Part

The hardest part wasn't writing Put() or Get().

Those operations are relatively straightforward.

The difficult part was defining what should happen when things go wrong.

Questions appeared everywhere:

What happens after a restart?

The index has to be reconstructed from the log.

What happens if a record is invalid?

The recovery process needs to detect it instead of silently producing incorrect state.

What happens when a key is deleted?

The delete needs to become part of the persistent history.

What happens with concurrent access?

The in-memory index and file operations need appropriate synchronization.

What happens as the log gets larger?

Append-only storage naturally creates a need for compaction.

These are the kinds of problems that existing database libraries normally solve for you.

Building VaultLog made those hidden engineering decisions visible.

What I Learned

The biggest lesson from this hackathon wasn't simply that you can build software without dependencies.

It was understanding why those dependencies exist in the first place.

A package like a database library isn't magic.

Underneath the API are decisions about:

• Storage layout
• Indexing
• Serialization
• Durability
• Recovery
• Concurrency
• Integrity

When you remove the package, those decisions become your responsibility.

That is what made the Zero Dependency constraint valuable.

What VaultLog Does Not Try to Be

VaultLog is deliberately small.

It is not intended to replace:

• PostgreSQL
• SQLite
• Distributed databases
• Production-scale storage systems

It currently has important limitations:

• The index is held in memory.
• The log grows until compaction is implemented.
• It is designed for embedded/single-process use.
• There is no distributed replication.
• There is no SQL query engine.

These aren't hidden.

Understanding the limitations is part of understanding the design.

Why Zero Dependencies?

Zero dependencies doesn't mean third-party packages are bad.

Libraries save enormous amounts of engineering time and are essential for many real-world systems.

The value of this challenge is different.

It forces you to understand the layer underneath the abstraction.

For VaultLog, that meant going from:

"Use a database package."

to:

"How does a database actually store a record?"

And then building that layer ourselves.

Final Thoughts

VaultLog started with a constraint:

No third-party runtime dependencies.

It ended up becoming an exploration of how a small storage engine actually works.

The result is a persistent key-value store with:

• Append-only storage
• In-memory indexing
• Restart recovery
• Integrity validation
• Concurrent access protection
• CLI operations
• Standard-library-only implementation

The most interesting part wasn't removing dependencies.

It was discovering what those dependencies were doing for us.

And that's probably the biggest takeaway from Zero Dependency:

Don't just use the abstraction. Understand the layer underneath it.

Built With

Go 1.27

Standard library only.

Third-party runtime dependencies: 0.

Project: VaultLog
Track: Data & Storage