SNKV- Key value store based on sqlite b-tree engine

hash-anu — Thu, 05 Mar 2026 10:52:49 +0000

There are many key--value stores available, such as RocksDB, LevelDB,
LMDB, etc. However, these often consume quite a lot of memory. When I
searched on Reddit, I found that whenever someone asks about using a
key--value store, people frequently suggest using SQLite.

Example discussion:
https://www.reddit.com/r/rust/comments/1ls5ynr/recommend_a\_keyvalue_store/

Even though SQLite is a SQL database, people often use it as a
key--value store. This made me want to develop my own key--value store
that uses SQLite's storage engine instead of the entire SQLite stack. AI
really helped me understand the lower layers of SQLite, such as the
B-tree, pager, and OS layers.

This led to the creation of:

https://github.com/hash-anu/snkv

Architecture

The architecture of SNKV is very simple:

kvstore layer -> b-tree layer -> pager -> os

The lower layers are already battle-tested and production-ready.

My task was to develop kvstore.c so that it consumes the APIs of the
B-tree layer while the lower layers work without any issues. In kvstore, I mostly used sqlite utility functions so that there are no third party library dependencies and every thing can be encapsulated into single file.

How to Use SNKV

C / C++

If you have a C/C++ project and want to use SNKV, it's very simple.

Generate snkv.h:

make snkv.h

Or download the ZIP from the 0.4.0 release.

Since it is a single-header kvstore, you can directly include it in
your project. In one of your .c or .cpp files add:

#define SNKV_IMPLEMENTATION

This ensures the implementation is compiled into your executable.

You can also review the API specification:

https://github.com/hash-anu/snkv/blob/master/docs/api/API_SPECIFICATION.md

Python

Python developers can simply install it using pip:

pip install snkv

Documentation for Python APIs:

https://github.com/hash-anu/snkv/blob/master/docs/python_api/API.md

Optimizations in KVStore

The entire SQL layer is bypassed. Data is stored in the format:

key_len | key | value

inside the B-tree.

This format allows fetching values based on key prefixes quickly in
O(log n) time.
Each table maintains a cached read cursor, improving the
performance of read operations such as get, exists, etc.
By default, a read transaction starts immediately after
kvstore_open. This reduces the time needed to find mxFrame
during read operations.

For write operations:

The read transaction is committed.
A write transaction runs and commits.
The read transaction starts again.

Examples

Examples demonstrating different SNKV use cases:

https://github.com/hash-anu/snkv/tree/master/python/examples
https://github.com/hash-anu/snkv/tree/master/examples

Crash Testing

A crash test was implemented that:

Writes deterministic key--value pairs into a 10 GB WAL-mode database
Forcefully kills the writer using SIGKILL during active writes
Verifies on restart that:

Results:

Every committed transaction exists with byte-exact values
No partial transactions are visible
The database shows zero corruption

Eco system support

Since storage engine of SNKV is same as Sqlite, tools which are relying on lower layers can be directly used with SNKV, I have verified tool such as

LiteFs
Wal based backup tools
Rollback journal tools

Next Step

Try SNKV and experiment with it.

If you encounter any issues or have suggestions, please open an issue:

https://github.com/hash-anu/snkv/issues

Feedback and thoughts are welcome.

Introducing AnuDB: A Lightweight, Serverless Document Database

hash-anu — Mon, 21 Apr 2025 07:05:26 +0000

Hey Dev.to community! 👋

I'm excited to share AnuDB, a project I've been working on that aims to solve document storage needs for C++ applications, particularly those running in embedded environments.

What is AnuDB?

AnuDB is a lightweight, serverless document database designed specifically for C++ applications. It offers efficient storage of JSON documents through MessagePack serialization, providing a serverless, schema-less solution for applications requiring flexible data management with robust query capabilities.

⭐ If you find this project useful, please consider giving it a star on GitHub! ⭐

Key Features

Embedded & Serverless: Run directly within your application with no separate server process
JSON Document Storage: Store and query complex JSON documents with rich query capabilities
RocksDB Backend: Built on RocksDB for high performance and durability
MQTT Interface: Connect and operate via MQTT protocol from various platforms
High Concurrency: Supports 32 concurrent worker threads for handling MQTT requests
Docker Support: Easy deployment with Docker

Ideal for AI and IoT Applications

AnuDB is particularly well-suited for AI and IoT domains:

Edge Computing: Store and process data directly on edge devices with limited resources
IoT Data Collection: Efficiently collect and query sensor data from distributed IoT networks
AI Model Results: Store inference results and model outputs in a structured, queryable format
Embedded Systems: Optimized for constrained environments common in IoT deployments
MQTT Integration: Seamlessly integrates with existing IoT MQTT infrastructure
Offline-First Applications: Support for disconnected operation in remote deployments

MQTT Interface - Connect from Anywhere

One of the features I'm most excited about is the MQTT interface, allowing you to interact with AnuDB from any platform that supports MQTT, without direct C++ integration.

Quick Start

Here's a simple example to get you started:

#include "Database.h"
#include <iostream>

int main() {
    anudb::Database db("./my_database");
    anudb::Status status = db.open();

    // Create a collection
    status = db.createCollection("users");
    anudb::Collection* users = db.getCollection("users");

    // Create a document
    nlohmann::json userData = {
        {"name", "Hash"},
        {"email", "hash@example.com"},
        {"age", 33}
    };
    anudb::Document doc("user001", userData);

    // Insert the document
    status = users->createDocument(doc);

    anudb::Document doc1;
    status = users->readDocument("user001", doc1);
    std::cout << doc1.data().dump(4) << std::endl;

    db.close();
    return 0;
}

Full Documentation

This post only scratches the surface of what AnuDB can do. For complete documentation including:

Detailed API reference
Query and update operations
Index management
MQTT command reference
Docker deployment options
Building from source
Performance considerations
Embedded platform optimizations

Check out the full README on GitHub: https://github.com/hash-anu/AnuDB

Get Involved

If you're interested in AnuDB, I'd love your feedback and contributions:

Star the repo: https://github.com/hash-anu/AnuDB
Try it out and file issues for any bugs or feature requests
Fork and submit PRs if you'd like to contribute code

Let me know in the comments what you think, or if you have any questions about using AnuDB in your projects!

DEV Community: hash-anu