Series: nosqlite devlog, Part 1
I do this thing sometimes where I get an idea at the wrong time. Not when I have free hours and a clear head, but at 1 AM, mid-week, while I should be doing something else entirely. The idea lands and refuses to leave.
The idea this time was simple on the surface: what if I built a database?
Not a production database. Not the next MongoDB or a distributed cloud thing. Just a small, honest, embedded database. One file, one process, document storage. Something I could actually finish understanding from top to bottom, because I was starting to realize that I had been using databases for years without really knowing what they were doing inside.
That bothered me.
So I started nosqlite. An experiment. A learning project. And what you are reading now is me trying to document how I built it, what I learned, and what it taught me about databases, data structures, systems programming, and Rust itself.
What Are We Actually Building
The description in the README is honest: "trying to build a small embedded nosql database from scratch in rust."
The name is a mashup of two real things: NoSQL and SQLite. Not because it is trying to replace either, but because it takes inspiration from both.
SQLite is one of the most deployed software libraries in human history. Every Android phone has it. Every iOS app probably uses it. Browsers use it. It is a self-contained database that lives in a single file. You do not start a server. You just open a file and start reading and writing. That simplicity is profound when you think about it. The whole database is just a file you can cp, mv, or put on a USB drive.
NoSQL is a broader idea. It basically means "not relational." Instead of tables with fixed columns, you store documents. A document is just a blob of structured data, something like JSON. The shape can vary per record. You do not declare a schema upfront. This model made a lot of sense for applications where data is fluid and hard to predict.
nosqlite tries to marry these two ideas. SQLite-like single file simplicity. NoSQL-like document storage.
That is the goal. And it is entirely an experiment.
Why I Built This in Rust
I was already shifting toward Rust as my primary language. I wrote about that decision separately. But this project cemented it.
Rust is the right tool for something like this because a database engine is fundamentally a systems problem. You are dealing with raw bytes. You are managing a file at the byte level. You are thinking about memory layout, about what happens when a process crashes mid-write, about how data is read back and verified. These are not higher-level concerns.
In TypeScript or Python, you would reach for a library to handle most of this. In Rust, you can actually go down to the level of reading four bytes at a known offset and interpreting them as a little-endian u32. You can describe the exact memory layout of a page. You can write a function that writes bytes into a buffer at a cursor position and have the type system enforce that you do not mess it up.
And the borrow checker, which every newcomer to Rust finds frustrating, becomes genuinely valuable when you are building something like this. You cannot accidentally have two mutable references to the same page buffer. The compiler forces you to think about ownership at every step, which turns out to be exactly what you want when you are building a storage engine.
Understanding What a Database Actually Is
Most developers, including me before this project, have a mental model of a database that looks something like: "you write a query, rows come back." That is the interface. The internals stay hidden.
Here is the simpler version of what actually happens.
A database is ultimately a program that manages a file. The file is structured in a very specific way so that the database can find things quickly, can guarantee that data is not corrupted, and can handle the program crashing at any moment without losing committed data.
Every database, whether it is SQLite or PostgreSQL, has roughly the same set of problems to solve:
How do you organize bytes in the file? You cannot just write raw data one record after another and hope for the best. You need structure. Databases use the concept of a page. A page is a fixed-size chunk of the file, usually 4096 bytes, which matches the OS's memory page size. The database reads and writes entire pages at once.
How do you find records? If every lookup required scanning the entire file, a database with millions of records would be unusably slow. This is where data structures like B-trees come in. B-trees let you find records in logarithmic time. nosqlite does not have a full B-tree index yet, but it is on the roadmap, and I will cover it in a later article.
How do you handle crashes? If you are in the middle of writing when the power goes out, your data should not become corrupt garbage. Most databases solve this with write-ahead logging, abbreviated WAL. You write your intention to a log first, then perform the actual write. On startup, you replay the log. nosqlite plans to implement WAL, but this is also future territory.
How do you support multiple users accessing the same database? This is the concurrency problem. nosqlite currently takes the simple path: only one process can open the database at a time, enforced with a lock file.
The Structure of nosqlite
The project is a Rust workspace with three crates:
nosqlite-core is the storage engine. Everything that reads from and writes to the database file lives here. The pager, the page layout, the catalog (which tracks collections), the document storage logic, the binary codec.
nosqlite-ffi is a C ABI bindings layer. It exposes the core engine through a C-compatible API so that other languages can use it. Go and Python bindings already exist. Dart/Flutter bindings are planned.
nosqlite-shell is a command-line shell. You can open a database file, create collections, insert documents, query them, and delete them, all from the terminal.
That layered design matters. The core does not know or care about what calls it. The FFI layer wraps it in a way that any language with a C FFI can use. The shell is just one consumer among many.
NoSQL Concepts You Need to Know
Before the next articles get into code, it is worth establishing the vocabulary.
Database is the top-level container. In nosqlite, it maps to a single file.
Collection is roughly equivalent to a table in a relational database, but without a fixed schema. In MongoDB, you would call this a collection. In nosqlite, that is the term too.
Document is a single record within a collection. A document is a structured blob of data. nosqlite stores documents in BSON format, which stands for Binary JSON. BSON is what MongoDB uses under the hood. It is like JSON but binary-encoded, which makes it faster to parse and more efficient to store.
ObjectId is a unique identifier automatically assigned to every document. It is a 12-byte value that encodes a timestamp, a machine identifier, and a counter. This is also borrowed from MongoDB.
A typical interaction with nosqlite looks like this through its shell:
create users
insert users { "name": "Priyanshu", "age": 22 }
find users
[
{ "_id": "...", "name": "Priyanshu", "age": 22 }
]
Simple. Intentionally simple. The focus is on understanding what happens under the hood when you type those commands.
The Experiment Mindset
I want to be honest about something. This project is not finished. It probably will not be "finished" in a traditional sense. I am building it as I learn, which means some commits introduce bugs, some commits fix them, and some commits just rename things because I changed my mind about the design. All of that is in the public history.
I am okay with that. This is not a portfolio piece trying to impress someone. It is a record of me figuring things out in public. I find that approach more useful than studying theory alone. When something breaks and I have to debug it at the byte level, I actually understand it. Reading about it in a book does not give me the same thing.
Each article in this series follows the same pattern: a feature lands, I write about it. What the problem was, what I built, what it taught me, what I am still not sure about.
The repository is at https://github.com/PriyanshuPz/nosqlite . The commit history is open. You can see exactly what changed and when.
Now here is the thing that kept me up when I started this project. I had a blank file. A completely empty .nsql file on disk. And I had to decide: what does byte zero mean? What is the very first thing I write? What does the engine look for when it opens that file tomorrow and needs to trust that it is valid?
That question sounds simple. It is not. And the answer to it is the entire next article.
Top comments (1)
What does nosqlite check before treating a lock file as stale after a process crash?