DEV Community: Priyanshu Verma

Storing Documents: From a JSON String to Bytes on Disk

Priyanshu Verma — Sun, 25 Oct 2026 10:40:07 +0000

Series: nosqlite devlog, Part 4

By the end of the third article, nosqlite had empty collections.

You could launch the shell, type create users, and the database would happily nod, update its catalog page, and save the collection name to disk. But a database that only stores collection names is just an expensive notepad. It is the architectural equivalent of putting empty cardboard boxes in a warehouse and calling it a logistics empire.

It was time to store actual data.

This was the part of the project I had been looking forward to the most, because this is where database theory stops being abstract and starts turning into raw binary operations. When someone uses a document database, they think in JSON. They type { "name": "Priyanshu", "age": 22 } and expect the database to tuck that object away safely on disk, keep it intact through computer reboots, and hand it back when asked.

Turning that simple expectation into bytes inside a 4096-byte page forced me to make some of the most critical design decisions in the entire project.

The Temptation of Raw JSON

When you sit down to build a document database from scratch, there is an obvious shortcut whispering in your ear: why not just store JSON strings?

Think about how easy that sounds. In Rust, you can add serde_json, call to_string(), and get a plain UTF-8 string. You write that string directly into the file. If you want to read it back, you read the string and parse it. It is human-readable, every language supports it, and you can inspect your database with a regular text editor.

It feels like the right move until you think about what a database actually does when it reads data.

Imagine you have a collection with ten thousand documents. Each document has fifty fields, and you want to find all users where the age field is greater than twenty.

If your documents are stored as raw JSON strings, the database has to parse the text of every single document from byte zero. It has to scan through characters, match curly braces, skip whitespace, interpret quotation marks, unescape special characters, and search for the characters "age". If you are scanning ten thousand documents, your CPU spends all its cycles doing string parsing instead of database work.

Then there is the data type problem. JSON is loose. When JSON says 42, is that an 8-bit integer, a 32-bit integer, a 64-bit integer, or a floating-point number? JSON does not specify. It just has numbers.

What if you want to store a binary payload, like a cryptographic hash or an image thumbnail? JSON has no native binary type. You have to encode the bytes as base64 strings, which inflates the size by thirty-three percent and wastes storage.

What if you want to store a timestamp? JSON has no date type. You store it as an ISO 8601 string like "2026-10-19T20:00:00Z", which takes twenty-four bytes and requires string parsing every time you want to compare two dates.

This is why the team behind MongoDB created BSON, which stands for Binary JSON.

Why BSON Changes the Game

BSON takes the flexibility of JSON and encodes it into a structured binary format.

Instead of writing text that has to be parsed character by character, BSON prefixes every single field with a type byte and a length.

Let me make this concrete. If a document has a string field named "name" with the value "Priyanshu", BSON does not write "name": "Priyanshu". It writes something closer to this:

First, a single type byte (0x02), which signals: the following value is a UTF-8 string.
Next, the field name as a null-terminated C string: name\0.
Next, a 4-byte integer recording the length of the string value: 10 (including the null terminator).
Finally, the string bytes themselves: Priyanshu\0.

Why does this matter so much for performance?

Because of the length prefix! If a reader is scanning a document looking for age, and it encounters the name field, it does not have to parse the string "Priyanshu". It reads the length prefix (10 bytes), adds 10 to its read cursor, and jumps over the string in a single CPU instruction.

Integers in BSON are stored as raw 4-byte or 8-byte little-endian numbers, not digit characters. Comparing two integers is a single CPU comparison, not string parsing. Binary blobs are stored as raw bytes with a length prefix, with zero base64 overhead.

The only downside is that you cannot open the database file in Notepad and read your documents. But for an embedded database engine, that is a trade I will make every single day. The shell handles formatting JSON for humans; the storage engine works in binary.

Rather than reinventing a binary serialization format, nosqlite uses the bson crate from crates.io. It is battle-tested, fast, and handles the full BSON specification.

The Twelve Bytes of an ObjectId

In a relational database, you usually have an auto-incrementing integer for your primary key: 1, 2, 3, and so on.

In a document database, that approach falls apart quickly. If two different worker threads or distributed clients insert documents at the same time, coordinating an auto-incrementing integer requires centralized locking.

Instead, document databases use an _id field containing an ObjectId.

When someone inserts a document into nosqlite without an _id, the engine generates one automatically. An ObjectId is exactly 12 bytes long. But those 12 bytes are not random noise. They are packed with information:

The first 4 bytes are a Unix timestamp representing seconds since 1970.
The next 5 bytes are a random value generated once when the process starts.
The final 3 bytes are an incrementing counter that starts at a random number.

The most elegant part of this design is the first 4 bytes. Because the Unix timestamp sits at the very beginning in big-endian order (most significant byte first), ObjectIds naturally sort chronologically!

If you take two ObjectIds and compare them byte by byte from left to right, the one created earlier will almost always sort before the one created later. This gives you time-ordered IDs for free. You can query documents by insertion time or extract the exact creation timestamp of any document directly from its ID without storing a separate created_at field.

The random 5 bytes ensure that two different processes or machines inserting at the exact same second will not generate conflicting IDs. The 3-byte counter ensures that a single process inserting thousands of documents within the same second generates unique IDs for every single one.

In Rust, generating a new ObjectId with the bson crate is a single line:

let id = bson::oid::ObjectId::new();

Simple on the surface, but mathematically brilliant underneath.

How Documents Live Inside Pages

In the second article, we established that nosqlite organizes its entire file into fixed 4096-byte pages.

A collection might have five documents, or it might have fifty thousand documents. A single 4096-byte page can hold only a few kilobytes of data. How do we store a collection that spans dozens or hundreds of pages?

We build a classic computer science data structure directly on disk: a singly linked list.

The first page of documents in a collection points to the second page. The second page points to the third page. The third page has its pointer set to 0, which signals that it is the end of the chain.

Here is what the header of every document page looks like:

// Document page layout:
// +-----------------------------------+
// | Generic Page Header               |   16 bytes (page number, page type)
// +-----------------------------------+
// | next_page (u64)                   |    8 bytes (pointer to next page in chain)
// +-----------------------------------+
// | document_count (u32)              |    4 bytes (active documents on this page)
// +-----------------------------------+
// | used_space (u32)                  |    4 bytes (bytes written in payload area)
// +-----------------------------------+
// | [records follow here]             |   up to 4064 bytes of document data
// +-----------------------------------+

And here are the offset constants from crates/nosqlite-core/src/document/page.rs:

pub const NEXT_PAGE_OFFSET: usize = PAGE_HEADER_SIZE;           // 16
pub const DOCUMENT_COUNT_OFFSET: usize = NEXT_PAGE_OFFSET + 8;  // 24
pub const USED_SPACE_OFFSET: usize = DOCUMENT_COUNT_OFFSET + 4; // 28
pub const DOCUMENT_HEADER_SIZE: usize = USED_SPACE_OFFSET + 4;  // 32

The first 16 bytes are the generic page header.
Bytes 16 to 23 hold next_page, which is an 8-byte page number pointing to the next document page in the collection.
Bytes 24 to 27 hold document_count, a 4-byte integer tracking how many active documents live on this specific page.
Bytes 28 to 31 hold used_space, a 4-byte integer tracking how many bytes of data have been written into the payload area so far.

Together, the header takes up exactly 32 bytes. Since a page is 4096 bytes, that leaves 4064 bytes of usable space for storing records.

Notice how important that used_space field is. When you want to insert a new document, you do not have to scan through the page to find where the free space begins. You just read the 4 bytes of used_space, add 32, and you immediately have the exact byte offset where the new record should be written. It turns finding the insertion point into an instant O(1) operation.

The Anatomy of a Record

Inside that 4064-byte payload region, records are packed one after another.

Each individual record has a small 5-byte header followed by the raw BSON document:

// Per-record layout:
// +-------------------------------+
// | deleted_flag (u8)             |    1 byte  (0 = active, 1 = deleted)
// +-------------------------------+
// | bson_size (u32)               |    4 bytes (length of the BSON payload)
// +-------------------------------+
// | bson_bytes...                 |    bson_size bytes of raw document data
// +-------------------------------+

pub const RECORD_DELETED_FLAG_SIZE: usize = 1;
pub const RECORD_BSON_SIZE: usize = 4;
pub const RECORD_HEADER_SIZE: usize = RECORD_DELETED_FLAG_SIZE + RECORD_BSON_SIZE; // 5

The first byte is the deleted_flag. If it is 0, the record is alive. If it is 1, the record has been deleted.

The next 4 bytes store bson_size as a little-endian unsigned 32-bit integer. This tells the reader exactly how many bytes to read for this document.

Why do we need bson_size right in the record header?

Because documents in a NoSQL database have variable lengths. One document might be 40 bytes. The next document might be 600 bytes. Without bson_size, the reader would have no idea where the current document ends and where the next record begins. Furthermore, if a record is marked as deleted, the reader can look at bson_size, skip forward by that exact number of bytes, and arrive at the next record header without wasting CPU cycles decoding dead BSON data.

The Insertion Pipeline: From Shell to Disk

Let us trace the complete path of an insertion. You sit at your terminal and run:

insert users { "name": "Priyanshu", "age": 22 }

Here is what happens inside nosqlite, step by step.

First, the shell parses the command. It extracts the collection name ("users") and the raw JSON text. It passes that JSON text to a conversion function that parses it into a bson::Document in memory.

Second, the engine inspects the document. Does it have an _id field? No. So the engine calls ObjectId::new(), generates a fresh 12-byte identifier, and inserts it into the document under the key _id.

Third, the engine serializes the document into binary BSON bytes using our codec helper:

pub fn encode_document(doc: &Document) -> Result<Vec<u8>> {
    let mut bytes = Vec::new();
    doc.to_writer(&mut bytes).map_err(Error::BsonEncode)?;
    Ok(bytes)
}

The method to_writer turns the in-memory document into a compact slice of bytes. Let us say this document serializes to 45 bytes.

Fourth, the engine looks up the "users" collection in the catalog. It reads the catalog entry and checks first_document_page.

If first_document_page is 0, this collection has never had a document before. The engine calls the pager to allocate a brand new page from disk. Let us say the pager allocates page 2. The engine initializes page 2 as a PageType::CollectionData page, sets next_page = 0, sets document_count = 0, and sets used_space = 0.

Fifth, the engine checks if there is enough space on page 2 for the new record.

The math is simple:

available_space = PAGE_SIZE - DOCUMENT_HEADER_SIZE - used_space
required_space  = RECORD_HEADER_SIZE + bson_bytes.len()

Plugging in our numbers:

available_space = 4096 - 32 - 0 = 4064 bytes
required_space  = 5 + 45 = 50 bytes

We have 4064 bytes available and only need 50. There is plenty of space.

Sixth, the engine writes the record into the page buffer starting at offset 32 + used_space:

// Write active flag
page.data[offset] = 0;
offset += 1;

// Write BSON size as little-endian u32
let size = bson_bytes.len() as u32;
page.data[offset..offset + 4].copy_from_slice(&size.to_le_bytes());
offset += 4;

// Write raw BSON bytes
page.data[offset..offset + bson_bytes.len()].copy_from_slice(&bson_bytes);

Seventh, the engine updates the page bookkeeping. document_count becomes 1. used_space increases by 50 bytes. The page is marked dirty and written back to disk.

Finally, because this was the very first document for the "users" collection, the engine updates the catalog entry on page 1 so that first_document_page points to page 2, and updates the catalog's document count.

What happens if a page does not have enough space for the new record?

The engine allocates a new page, updates the existing page's next_page field to point to the new page number, writes the existing page to disk, and then writes the new record onto the newly allocated page. The on-disk linked list grows by one node.

Reading Data: The Honest Linear Scan

When you run find users in the shell, the engine reverses that pipeline.

It reads the catalog to find first_document_page. It loads that page into memory. It sets a cursor to offset 32 (where the first record starts) and walks forward through the payload.

For every record, it reads the 1-byte deleted flag. If the flag is 0, it reads the 4-byte bson_size, slices out the BSON bytes, and decodes them into a document:

pub fn decode_document(bytes: &[u8]) -> Result<Document> {
    Document::from_reader(bytes).map_err(Error::BsonDecode)
}

The shell takes that document, converts it to readable JSON, and displays it in the terminal.

The cursor advances by 5 + bson_size. When the cursor reaches the end of the used space on the page, the engine checks next_page. If next_page is not 0, it loads the next page in the chain and repeats the process. When next_page is 0, the scan is complete.

This is a linear scan. Every record on every page in the collection is inspected.

There is no B-tree index. There is no hash map shortcut. If you have ten documents, it is instantaneous. If you have ten million documents, it will take several seconds.

And I love that this is how it works right now. Because when you build a database from scratch, starting with an honest linear scan teaches you why indexes exist in the first place. An index is not magic; it is simply a separate data structure built to avoid this exact linear scan. We will get to indexes in a later article.

Soft Deletion and the Concept of Tombstones

When someone runs delete users <id>, what should the database do?

If you were working with a Vec<u8> in RAM, your first thought might be to delete the bytes and slide everything after it to the left, just like we did in the catalog.

But on a data page, that is a terrible idea.

First, performance. If a document page holds forty documents, and you delete the second document, you would have to shift thousands of bytes of memory to the left, update the offsets, and recalculate used_space.

Second, crash safety. What happens if the program crashes or the power cuts out right in the middle of sliding those three thousand bytes? Half your page is shifted, half is unshifted, and your database file is irreparably corrupted.

Third, concurrency. If a background query is reading documents while a write operation shifts memory beneath its feet, the reader will read scrambled garbage.

Instead, nosqlite uses soft deletion, often called tombstoning:

// Mark the record as deleted
page.data[record_offset] = 1;

To delete a document, we write a single byte with the value 1 into the record's deleted_flag. We decrement the page's document_count, decrement the catalog's document count, and flush the page.

That is it. The actual BSON document bytes remain on disk. They do not move. But the next time a reader scans the page, it sees deleted_flag == 1, notes the bson_size, skips right over the dead bytes, and moves to the next record.

The trade-off is fragmentation. If you insert a hundred documents and delete ninety of them, those ninety dead records still consume space on disk. The page's used_space counter does not decrease.

In production databases, this is solved by a process called compaction, where the database rewrites pages in the background, copying only live records to fresh pages and reclaiming the free space. Compaction is on the roadmap for nosqlite, but for a learning project, soft deletion is the cleanest, safest, and most robust starting point.

What We Built

Take a step back and look at what we have now.

We started in Article 1 with an idea: build a single-file document database in Rust.
In Article 2, we formatted a file with 4096-byte pages and magic bytes.
In Article 3, we built a catalog page to track collections and added crash-safe locking.
And now, in Article 4, we have a fully functional document storage pipeline.

You can open nosqlite, create collections, insert arbitrary JSON documents, have them stored as structured binary BSON on disk, scan them back out, and safely delete them using tombstones. If you close the shell and turn off your computer, the data will still be there tomorrow morning.

It is a real database.

But right now, it only exists as a Rust library and a command-line tool. If someone wants to build a web backend in Go, a data science pipeline in Python, or a mobile app in Flutter, they cannot use nosqlite. Rust code cannot simply be imported into another language.

To make nosqlite accessible to the rest of the programming world, we need a bridge. We need an interface that every programming language on earth understands: the C Application Binary Interface.

In the next article, we dive into the FFI layer. We will look at how to expose Rust structs as opaque C pointers, how to pass strings and error codes across language boundaries safely, and how to stop Rust panics from crashing a foreign host program.

Collections and the Catalog: How a Database Remembers What Exists

Priyanshu Verma — Thu, 22 Oct 2026 01:51:14 +0000

Series: nosqlite devlog, Part 3

Once I had page zero written to disk with those magic bytes NOSQLITE_DB___P8, I had a brief moment of satisfaction. I opened a hex editor, saw my custom signature staring back at me, and felt like I was actually getting somewhere.

Then reality set in.

I had an empty file with a header and absolutely no way to store anything useful yet. If you open a database, what is the very first thing you do? You create a collection. In SQL, you write CREATE TABLE. In MongoDB, you just start writing to a collection or create one explicitly. But what actually is a collection when you are sitting in front of a raw binary file?

It is not a folder on your hard drive. It is not some magical in-memory table. At the byte level, a collection is just a string name tied to a starting page number somewhere inside that single file.

My first lazy thought when planning this was tempting: why not just keep a HashMap<String, Vec<Document>> in memory and dump it to disk as a JSON blob when the program exits?

That would have taken an hour to write. But that is what you do if you are building a toy in Python or Node, and it completely misses the point of this experiment. If the power cuts out, your in-memory map is gone. If you copy the database file to another computer, that catalog has to travel with it inside the exact same file. The file has to be completely self-contained. It has to know its own collections, where they live, and how many documents they hold.

That meant building a catalog page.

The Table of Contents for Your Database

Think about a book. Before you read chapter three, you flip to the table of contents. The table of contents does not contain the story. It just tells you: Chapter 3 starts on page 45.

The catalog in nosqlite is literally just that table of contents.

For every collection you create, the engine needs to remember exactly three things:
What is the collection's name?
What page number does its first document live on?
How many documents are currently inside it?

That is all. If you know those three things, you can find the data, display counts in the shell, and know where to start scanning.

Where does this table of contents live? It gets its own dedicated page in the file. Specifically, it lives on page 1, sitting right after the file header. If you remember from the previous article, the file header on page 0 stores a field called catalog_page that holds the value 1. When the database boots up, it reads page 0, checks catalog_page, and immediately loads page 1 into memory.

The Anatomy of the Catalog Page

Let us look at how page 1 is laid out in memory and on disk.

// Catalog page layout:
//
// +-------------------------------+
// | Page Header                   |   16 bytes (generic header on every page)
// +-------------------------------+
// | collection_count (u32)        |    4 bytes (how many collections exist)
// +-------------------------------+
// | entries...                    |   packed one after another
// +-------------------------------+
//
// Each entry looks like this:
//
// +-------------------------------+
// | collection_name (\0)          |   variable length, null-terminated
// +-------------------------------+
// | first_document_page (u64)     |    8 bytes
// +-------------------------------+
// | document_count (u64)          |    8 bytes
// +-------------------------------+

The first 16 bytes belong to the generic page header that every single page in nosqlite has. That header records the page number, the page type (in this case, PageType::CollectionCatalog), and a couple of reserved bytes.

Right after those 16 bytes, we store 4 bytes for collection_count. This is an unsigned 32-bit integer that tracks how many collections currently exist in the database.

Why do we need this count? Because without it, a loop reading through the file would have no idea when to stop. It would keep reading random bytes until it crashed or hit corrupted memory. With collection_count, the reader knows upfront: there are 3 collections here, so read exactly 3 entries and then stop.

The catalog header size is therefore 20 bytes:

pub const CATALOG_HEADER_SIZE: usize = PAGE_HEADER_SIZE + 4; // 16 + 4 = 20

Since a page is 4096 bytes total, that leaves 4076 bytes for collection entries. In an embedded database where you usually have five, ten, or maybe thirty collections, 4076 bytes is plenty of room.

Null Terminators and C Strings

Here is the struct that represents a collection entry in Rust:

#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct CatalogEntry {
    pub name: String,
    pub first_document_page: u64,
    pub document_count: u64,
}

In Rust, this looks clean and friendly. But when this struct gets serialized into raw bytes on disk, we have to make an important choice: how do we store the string name?

In modern high-level languages, a string always carries its length with it. But at the raw binary level, there are two classic approaches. You can either use a Pascal string, where you write a length byte first and then the characters, or you can use a null-terminated C string, where you write the characters and end with a literal zero byte (\0).

I chose the null-terminated C string approach.

When you create a collection named "users", the engine writes the ASCII bytes for u, s, e, r, s, followed by 0x00. In hex, that is 75 73 65 72 73 00. That zero byte acts as a red light for the reader. The reader walks forward byte by byte, and the instant it sees a zero, it knows the name is finished.

Right after that zero byte, we write 8 bytes for first_document_page and 8 bytes for document_count, both encoded as little-endian unsigned 64-bit integers.

This brings up a fascinating detail about variable-length binary layouts. Because collection names can have different lengths, the entries do not have a fixed size. A collection named "users" takes 6 bytes for the name plus 16 bytes for the two u64 numbers, totaling 22 bytes. A collection named "audit_logs_v2" takes 14 bytes for the name plus 16 bytes for the numbers, totaling 30 bytes.

Because of this, you cannot simply calculate an offset with math like index * entry_size. You cannot jump directly to the fifth collection. You have to start at byte 20 and walk forward with a cursor, reading each entry to know where the next one begins.

Why the Catalog Struct Is Completely Empty

When you look at the source code in crates/nosqlite-core/src/catalog/catalog.rs, you find something that might look bizarre at first:

pub struct Catalog;

The struct has no fields. It holds no data in memory. It is a zero-sized type.

Every single method on Catalog takes a reference to a Page or Pager and acts directly on the byte buffer:

impl Catalog {
    pub fn initialize(page: &mut Page) {
        page.set_page_type(PageType::CollectionCatalog);

        // collection_count = 0
        page.data[PAGE_HEADER_SIZE..PAGE_HEADER_SIZE + 4]
            .copy_from_slice(&0u32.to_le_bytes());
    }
}

Let us look closely at what 0u32.to_le_bytes() is doing here, because this is fundamental to how systems programming works.

The number 0 as an unsigned 32-bit integer occupies 4 bytes. The method to_le_bytes() converts that number into a 4-byte array in little-endian order, where the least significant byte comes first. For zero, that array is just [0, 0, 0, 0]. If the count were 1, it would be [1, 0, 0, 0].

The call to copy_from_slice takes those 4 bytes and copies them directly into page.data starting at offset 16. That is how the collection count is set to zero when the database is first formatted.

Why did I make Catalog stateless instead of storing a Vec<CatalogEntry> inside it?

Because holding an in-memory copy of the catalog creates two sources of truth. What happens if your in-memory list says there are three collections, but a disk write failed and the file only has two? Now your program is in an inconsistent state, and debugging that will make you question your career choices.

By making Catalog stateless, the page itself is always the single source of truth. Every time we need to create, delete, or list collections, we read the page, do the work, and flush the page. There is zero risk of in-memory cache drift.

Creating a Collection: The Journey of a Write

When you type create users in the shell, here is the exact journey that command takes through the engine.

First, the engine calls the pager to load the catalog page into memory. The pager gives back a 4096-byte mutable slice of memory representing page 1.

The engine reads the first 4 bytes after the page header to find out how many collections already exist. Let us say collection_count is 2.

Next, it has to make sure we are not creating a duplicate. It sets a cursor variable to offset 20 (the start of the entries). It loops twice, reading each existing collection name using a helper function called read_cstring. If any name matches "users", it stops immediately and returns an error. We do not allow duplicate collections.

If no duplicate is found, the cursor is now pointing to the exact byte where the previous entry ended. That is our insertion point.

The engine checks whether there is enough space left on page 1 for the new entry. The new entry needs 6 bytes for "users\0", 8 bytes for the page pointer, and 8 bytes for the document count, which is 22 bytes. If the page has enough room, it writes the string bytes, writes 8 bytes of zero for first_document_page (since a new collection has no documents yet), and writes 8 bytes of zero for document_count.

Then it goes back to offset 16, increments the collection_count from 2 to 3, and writes the new count back to the page buffer.

Finally, the pager marks the page as dirty and flushes it to disk.

Notice how simple and mechanical that is. There is no query planner, no schema validator, no background worker. It is just reading a small buffer, checking for duplicates, appending 22 bytes, incrementing a counter, and writing the page back.

Listing Collections with a Cursor

Reading the list of collections reverses that entire process:

pub fn list(pager: &mut Pager) -> Result<Vec<CatalogEntry>> {
    let page = pager.read_page(pager.header.catalog_page)?;
    let count = Self::collection_count(&page)?;
    let mut cursor = CATALOG_HEADER_SIZE;
    let mut entries = Vec::new();

    for _ in 0..count {
        let name = read_cstring(&page.data, &mut cursor)?;
        let first_document_page = read_u64(&page.data, &mut cursor);
        let document_count = read_u64(&page.data, &mut cursor);

        entries.push(CatalogEntry {
            name,
            first_document_page,
            document_count,
        });
    }

    Ok(entries)
}

The cursor pattern is one of the most useful patterns in binary parsing. A cursor is simply a usize variable that keeps track of where we are currently reading in the byte slice.

When read_cstring(&page.data, &mut cursor) runs, it starts at cursor, reads characters until it hits 0x00, constructs a Rust String, and advances cursor past the zero byte.

Then read_u64 reads the next 8 bytes as a little-endian number and advances cursor by 8. Then it does it again for document_count.

After looping count times, the function returns a clean Vec<CatalogEntry> ready for the shell to print.

Now, someone who took a computer science algorithms class might look at this and say: wait, that is an O(n) linear scan. Why didn't you build a B-tree or a hash map for the catalog?

Because context matters. The entire catalog page is 4096 bytes. That is 4 kilobytes. A modern CPU can scan 4KB of contiguous memory in its L1 cache in a fraction of a microsecond. If you have 20 collections, an O(n) linear scan takes virtually zero time. Building a complex B-tree just to store 20 collection names would be textbook over-engineering. Never pay for complexity you do not actually need.

Deleting a Collection: The Shift-Left Trick

Deleting a collection is where the byte manipulation gets interesting.

When someone deletes a collection, you cannot simply write zeros over its entry and leave a hole in the middle of the page. If the reader encounters a gap of zeros, it will either think the collection name is an empty string or get confused about where the next entry begins. The entries must always remain packed tightly together.

To solve this, I used a shift-left strategy.

Imagine a row of physical books sitting on a shelf. If you remove the book in the middle, you get an empty gap. What do you do? You slide all the books to the right of that gap one position to the left. The gap disappears, and all the books are neatly packed again.

In Rust, the standard library provides a method called copy_within on slices that does exactly this:

page.data.copy_within(entry_end..used_end, entry_start);

copy_within takes a source range of bytes and copies them to a new destination offset inside the exact same slice. It safely handles overlapping memory regions, which is crucial because when you slide bytes to the left, the source and destination overlap.

After sliding the bytes left, we decrement collection_count by 1.

What happens to the leftover bytes at the far right of the entries region? Nothing! We do not even need to zero them out. Because collection_count is decremented, the reader will stop reading earlier and will never look at those leftover bytes. They will simply be overwritten the next time a new collection is created.

When a collection is deleted, we also free any document pages that belonged to it so they can be reused later. We will see how page reuse works when we talk about the free-page list.

Preventing Catastrophe with a Lock File

Above the Catalog and the Pager sits the Database struct. This is the main entry point that the shell and external applications interact with:

pub struct Database {
    pager: Pager,
    lock_path: PathBuf,
    _lock_file: File,
}

Notice that field named _lock_file with the leading underscore.

In Rust, if you declare a variable or a field and never call any methods on it, the compiler will yell at you about an unused variable. The underscore tells the compiler: I know this looks unused, but please leave it alone.

And leaving it alone is the whole point!

When Database::open is called, it creates a small file next to the database with a .lock extension. It opens that file and stores the open file handle inside _lock_file.

As long as the Database instance exists in your program, that file handle remains open. When your program finishes or the Database goes out of scope, Rust drops the struct. Rust's ownership system automatically closes the file handle, and the operating system releases the lock.

This is the classic RAII pattern, which stands for Resource Acquisition Is Initialization. You acquire the resource on creation, and cleanup happens automatically on destruction. You do not have to remember to call a close() method.

But what happens if the power cord is pulled out of the wall while the database is running?

If the system crashes violently, the .lock file remains sitting on the hard drive even though the process is dead. If another program tries to open the database later, it sees the .lock file and thinks the database is still in use.

To solve this, nosqlite checks for stale locks on startup:

pub fn open(path: impl AsRef<Path>) -> Result<Self> {
    let lock_path = path.with_extension("lock");

    if lock_path.exists() {
        let stale = match read_lock_file(&lock_path) {
            Ok(lock) => !process_exists(lock.pid),
            Err(_) => true,
        };

        if stale {
            eprintln!("Removing stale lock file");
            let _ = std::fs::remove_file(&lock_path);
        } else {
            return Err(Error::DatabaseLocked);
        }
    }
    // ... proceed with opening
}

When nosqlite creates a lock file, it writes the current process ID into it. On startup, if a lock file exists, the engine reads that PID and asks the operating system: is a process with this ID currently running?

If the answer is no, the previous process died without cleaning up. The lock is stale. nosqlite deletes the file and continues opening normally. If the process is still running, the database is genuinely in use by another instance, and the engine refuses to open to prevent data corruption.

It is a simple, pragmatic mechanism that prevents two processes from destroying the same database file while handling unexpected crashes gracefully.

Where We Stand

At this point in the project, nosqlite had a solid foundation.

It could format a file, stamp its magic bytes, maintain a catalog of collections, slide bytes around safely during deletions, and protect the file from concurrent access with process locking.

You could open the shell, run create users, run create orders, and run list. The database would print:

users: 0 documents
orders: 0 documents

Everything worked. But every single collection had first_document_page set to 0.

It was like having a filing cabinet with neatly labeled folders, but every single folder was completely empty. We had built the entire organizational structure of a database without storing a single byte of actual user data.

That changes in the next article.

In Part 4, we finally put real data into those folders. We take a JSON string typed into the terminal, break down why storing raw JSON is a terrible idea, convert it into binary BSON format, allocate document pages, link them together in a chain, and write our very first document to disk.

The File Format: Pages, Headers, and Magic Bytes

Priyanshu Verma — Mon, 19 Oct 2026 15:27:59 +0000

Series: nosqlite devlog, Part 2

The first article was about the why. This one is where things get concrete.

I remember the exact moment I opened a hex editor on a SQLite file for the first time. The first 16 bytes said 53 51 4C 69 74 65 20 66 6F 72 6D 61 74 20 33 00. That decodes to SQLite format 3. A literal string. Right there at byte zero. Before any data, before any tables, before anything useful, the file just tells you what it is.

I thought that was brilliant. And I knew immediately that nosqlite would need the same thing.

When you open a nosqlite database for the first time, the file does not exist yet. The code creates it. But it does not just create a blank file and start dumping data. It sets up a structure. A layout. A way of knowing where everything is and what it means.

This is the file format.

Why Structure Matters

Imagine a file that contains two documents, both stored as raw BSON bytes, one after another. How do you know where the first document ends and the second begins? How do you know if the file is a valid nosqlite database and not a corrupted archive or a random binary blob? How do you find a specific document without reading everything from byte zero?

You cannot answer any of those questions without structure.

Every database engine, from SQLite to PostgreSQL to LevelDB, has a defined file format. It is as much a part of the project as the code itself. nosqlite is no different.

The Page: The Fundamental Unit

The core abstraction in nosqlite is the page. A page is a fixed-size block of bytes, exactly 4096 bytes. This number is not arbitrary.

Most modern operating systems use a memory page size of 4096 bytes. When you read a file, the OS reads it in chunks that align to this boundary. If your database reads in 4096-byte units, you are playing along with the OS instead of fighting it. One read operation brings in exactly one page.

Here is how the page is defined in the code:

pub const PAGE_SIZE: usize = 4096;
pub const PAGE_HEADER_SIZE: usize = 16;

#[repr(u8)]
pub enum PageType {
    Free = 0,
    CollectionCatalog = 1,
    CollectionData = 2,
}

Every page has a header that occupies the first 16 bytes. The rest of the page is available for data. The page type tells the engine what kind of data is in this page. Right now there are three types: a free page (not in use), a catalog page (which tracks collections), and a data page (which holds actual documents).

That enum attribute #[repr(u8)] is a Rust thing worth understanding. It tells the compiler to represent the enum as a single byte. This matters because this value gets written directly into the file. When you read that byte back, you can determine what kind of page you are looking at.

The File Header

The very first thing in a nosqlite database file is the file header. It lives in the first 4096 bytes, which is a full page. Even though it does not fill all 4096 bytes, it reserves the entire first page so that page numbering is consistent. Page 0 is always the header.

pub const FILE_HEADER_SIZE: u64 = 4096;
pub const MAGIC_BYTES: &[u8; 16] = b"NOSQLITE_DB___P8";

#[derive(Debug, Clone)]
pub struct FileHeader {
    pub version: u32,
    pub total_pages: u64,
    pub free_list_head: u64,
    pub catalog_page: u64,
}

Take a look at MAGIC_BYTES. This 16-byte sequence is the first thing written into every nosqlite database file. When you open a file, the very first check is: do these first 16 bytes match? If they do not, this is not a nosqlite database, and the engine refuses to proceed.

This is called a magic number or magic bytes. Every well-designed binary format has one. PNG files start with \x89PNG\r\n\x1a\n. ZIP files start with PK. SQLite files start with SQLite format 3. The magic bytes let any reader quickly identify the format before attempting to parse anything.

The P8 at the end is personal. It is a nod to p8labs, my own project namespace. Every file this engine creates carries that marker.

What the Header Tracks

The FileHeader struct holds four pieces of information:

version is a u32 that identifies the format version. This is future-proofing. If the file format changes in a breaking way, the version number changes too. An older reader can detect that the format is newer than it understands and refuse gracefully.

total_pages is a u64 counting how many pages exist in the file right now. Every time a new page is allocated, this counter increments.

free_list_head is a u64 holding the page number of the first free page. When you delete a collection or its documents, those pages do not disappear from the file. They go onto a linked list of free pages. Next time you need a new page, you can reuse one from this list instead of growing the file. This is 0 when there are no free pages.

catalog_page is a u64 holding the page number of the catalog page. The catalog is where the list of all collections is stored. You need to know where it is to do anything useful.

Binary Encoding

The file header and every other structure in nosqlite is encoded in little-endian format. Little-endian means the least-significant byte comes first.

For example, the number 1 as a u32 in little-endian is 01 00 00 00. In big-endian it would be 00 00 00 01. Most consumer hardware (x86, ARM in modern Linux defaults) is little-endian, which is why most database formats choose it.

The codec module handles reading and writing these values:

pub fn read_u32(buffer: &[u8], cursor: &mut usize) -> Result<u32> {
    let bytes = buffer.get(*cursor..*cursor + 4).ok_or(Error::Corruption)?;
    *cursor += 4;
    Ok(u32::from_le_bytes(
        bytes.try_into().map_err(|_| Error::Corruption)?,
    ))
}

pub fn read_u64(buffer: &[u8], cursor: &mut usize) -> Result<u64> {
    let bytes = buffer.get(*cursor..*cursor + 8).ok_or(Error::Corruption)?;
    *cursor += 8;
    Ok(u64::from_le_bytes(
        bytes.try_into().map_err(|_| Error::Corruption)?,
    ))
}

The cursor is a mutable reference to a position. Each read_* call advances the cursor by the number of bytes consumed. This pattern makes sequential reading clean. You start the cursor at 0, call read_u16, then read_u32, then read_u64, and the cursor tracks your position through the buffer automatically. No manual offset arithmetic.

The ok_or(Error::Corruption) at the end is the defensive part. If the buffer is shorter than expected, the engine returns a Corruption error instead of panicking. In a storage engine, unexpected short reads almost always mean the file has been truncated or corrupted. Detecting this rather than crashing is the correct behavior.

The Pager

The structure that manages all page reads and writes is called the Pager. This is a concept borrowed directly from SQLite's architecture.

pub struct Pager {
    file: File,
    pub header: FileHeader,
}

impl Pager {
    pub fn open(path: impl AsRef<Path>) -> Result<Self> {
        let mut file = OpenOptions::new()
            .create(true)
            .read(true)
            .write(true)
            .open(path)?;

        let header = if file.metadata()?.len() == 0 {
            Self::initialize_database(&mut file)?
        } else {
            read_header(&mut file)?
        };
        Ok(Self { file, header })
    }
    // ...
}

When you call Pager::open, it either initializes a fresh database or reads the existing header. The check is simple: if the file length is 0, the file is brand new and needs to be set up. Otherwise, read the header and trust that the rest of the file is structured correctly.

The OpenOptions call is standard Rust file I/O. create(true) means create the file if it does not exist. read(true) and write(true) give the handle both read and write permissions.

The Pager is the only thing in the codebase that directly interacts with the file. Every other module that needs to read or write pages goes through the Pager. This is important for correctness. Centralizing file access means page reads and writes are consistent. In the future, caching pages in memory before writing them to disk will happen inside the Pager, without touching anything else.

Bootstrap: First Database Write

When initialize_database runs on a fresh file, it does a few things:

It writes the magic bytes at byte 0. It writes an initial FileHeader. It allocates the first real page (page 1) as the catalog page and initializes it. It updates the header to point catalog_page at 1. Then it flushes everything to disk.

After that, the file is a valid nosqlite database with no collections and no documents, but with the infrastructure in place to hold them.

On every subsequent open, the Pager reads the header, verifies the magic bytes, and trusts the structure. Nothing needs to be rebuilt.

The File Layout So Far

Here is what the file looks like after initialization:

Byte 0       : NOSQLITE_DB___P8   (16 bytes, magic)
Byte 16      : version             (u32, 4 bytes)
Byte 20      : total_pages         (u64, 8 bytes)
Byte 28      : free_list_head      (u64, 8 bytes)
Byte 36      : catalog_page        (u64, 8 bytes)
Byte 44-4095 : padding / reserved
Byte 4096    : Catalog page header (16 bytes)
Byte 4112+   : Catalog page data

This is the foundation. Every feature built on top of nosqlite reads and writes through this structure.

In the next article, we look at collections. What is a collection in a NoSQL database? How does the catalog page know what collections exist and where their documents live? What happens at the byte level when you create a collection or delete one?

That is where the data starts to feel real.

Building a NoSQL Database in Rust, from Scratch: The Why

Priyanshu Verma — Sat, 17 Oct 2026 14:27:27 +0000

Series: nosqlite devlog, Part 1

I do this thing sometimes where I get an idea at the wrong time. Not when I have free hours and a clear head, but at 1 AM, mid-week, while I should be doing something else entirely. The idea lands and refuses to leave.

The idea this time was simple on the surface: what if I built a database?

Not a production database. Not the next MongoDB or a distributed cloud thing. Just a small, honest, embedded database. One file, one process, document storage. Something I could actually finish understanding from top to bottom, because I was starting to realize that I had been using databases for years without really knowing what they were doing inside.

That bothered me.

So I started nosqlite. An experiment. A learning project. And what you are reading now is me trying to document how I built it, what I learned, and what it taught me about databases, data structures, systems programming, and Rust itself.

What Are We Actually Building

The description in the README is honest: "trying to build a small embedded nosql database from scratch in rust."

The name is a mashup of two real things: NoSQL and SQLite. Not because it is trying to replace either, but because it takes inspiration from both.

SQLite is one of the most deployed software libraries in human history. Every Android phone has it. Every iOS app probably uses it. Browsers use it. It is a self-contained database that lives in a single file. You do not start a server. You just open a file and start reading and writing. That simplicity is profound when you think about it. The whole database is just a file you can cp, mv, or put on a USB drive.

NoSQL is a broader idea. It basically means "not relational." Instead of tables with fixed columns, you store documents. A document is just a blob of structured data, something like JSON. The shape can vary per record. You do not declare a schema upfront. This model made a lot of sense for applications where data is fluid and hard to predict.

nosqlite tries to marry these two ideas. SQLite-like single file simplicity. NoSQL-like document storage.

That is the goal. And it is entirely an experiment.

Why I Built This in Rust

I was already shifting toward Rust as my primary language. I wrote about that decision separately. But this project cemented it.

Rust is the right tool for something like this because a database engine is fundamentally a systems problem. You are dealing with raw bytes. You are managing a file at the byte level. You are thinking about memory layout, about what happens when a process crashes mid-write, about how data is read back and verified. These are not higher-level concerns.

In TypeScript or Python, you would reach for a library to handle most of this. In Rust, you can actually go down to the level of reading four bytes at a known offset and interpreting them as a little-endian u32. You can describe the exact memory layout of a page. You can write a function that writes bytes into a buffer at a cursor position and have the type system enforce that you do not mess it up.

And the borrow checker, which every newcomer to Rust finds frustrating, becomes genuinely valuable when you are building something like this. You cannot accidentally have two mutable references to the same page buffer. The compiler forces you to think about ownership at every step, which turns out to be exactly what you want when you are building a storage engine.

Understanding What a Database Actually Is

Most developers, including me before this project, have a mental model of a database that looks something like: "you write a query, rows come back." That is the interface. The internals stay hidden.

Here is the simpler version of what actually happens.

A database is ultimately a program that manages a file. The file is structured in a very specific way so that the database can find things quickly, can guarantee that data is not corrupted, and can handle the program crashing at any moment without losing committed data.

Every database, whether it is SQLite or PostgreSQL, has roughly the same set of problems to solve:

How do you organize bytes in the file? You cannot just write raw data one record after another and hope for the best. You need structure. Databases use the concept of a page. A page is a fixed-size chunk of the file, usually 4096 bytes, which matches the OS's memory page size. The database reads and writes entire pages at once.

How do you find records? If every lookup required scanning the entire file, a database with millions of records would be unusably slow. This is where data structures like B-trees come in. B-trees let you find records in logarithmic time. nosqlite does not have a full B-tree index yet, but it is on the roadmap, and I will cover it in a later article.

How do you handle crashes? If you are in the middle of writing when the power goes out, your data should not become corrupt garbage. Most databases solve this with write-ahead logging, abbreviated WAL. You write your intention to a log first, then perform the actual write. On startup, you replay the log. nosqlite plans to implement WAL, but this is also future territory.

How do you support multiple users accessing the same database? This is the concurrency problem. nosqlite currently takes the simple path: only one process can open the database at a time, enforced with a lock file.

The Structure of nosqlite

The project is a Rust workspace with three crates:

nosqlite-core is the storage engine. Everything that reads from and writes to the database file lives here. The pager, the page layout, the catalog (which tracks collections), the document storage logic, the binary codec.

nosqlite-ffi is a C ABI bindings layer. It exposes the core engine through a C-compatible API so that other languages can use it. Go and Python bindings already exist. Dart/Flutter bindings are planned.

nosqlite-shell is a command-line shell. You can open a database file, create collections, insert documents, query them, and delete them, all from the terminal.

That layered design matters. The core does not know or care about what calls it. The FFI layer wraps it in a way that any language with a C FFI can use. The shell is just one consumer among many.

NoSQL Concepts You Need to Know

Before the next articles get into code, it is worth establishing the vocabulary.

Database is the top-level container. In nosqlite, it maps to a single file.

Collection is roughly equivalent to a table in a relational database, but without a fixed schema. In MongoDB, you would call this a collection. In nosqlite, that is the term too.

Document is a single record within a collection. A document is a structured blob of data. nosqlite stores documents in BSON format, which stands for Binary JSON. BSON is what MongoDB uses under the hood. It is like JSON but binary-encoded, which makes it faster to parse and more efficient to store.

ObjectId is a unique identifier automatically assigned to every document. It is a 12-byte value that encodes a timestamp, a machine identifier, and a counter. This is also borrowed from MongoDB.

A typical interaction with nosqlite looks like this through its shell:

create users
insert users { "name": "Priyanshu", "age": 22 }
find users
[
{ "_id": "...", "name": "Priyanshu", "age": 22 }
]

Simple. Intentionally simple. The focus is on understanding what happens under the hood when you type those commands.

The Experiment Mindset

I want to be honest about something. This project is not finished. It probably will not be "finished" in a traditional sense. I am building it as I learn, which means some commits introduce bugs, some commits fix them, and some commits just rename things because I changed my mind about the design. All of that is in the public history.

I am okay with that. This is not a portfolio piece trying to impress someone. It is a record of me figuring things out in public. I find that approach more useful than studying theory alone. When something breaks and I have to debug it at the byte level, I actually understand it. Reading about it in a book does not give me the same thing.

Each article in this series follows the same pattern: a feature lands, I write about it. What the problem was, what I built, what it taught me, what I am still not sure about.

The repository is at https://github.com/PriyanshuPz/nosqlite . The commit history is open. You can see exactly what changed and when.

Now here is the thing that kept me up when I started this project. I had a blank file. A completely empty .nsql file on disk. And I had to decide: what does byte zero mean? What is the very first thing I write? What does the engine look for when it opens that file tomorrow and needs to trust that it is valid?

That question sounds simple. It is not. And the answer to it is the entire next article.

How to Read Open Source Code, Python Edition

Priyanshu Verma — Fri, 19 Jun 2026 05:25:15 +0000

Hi,

Today I was reading an open-source project that I found interesting, and it made me realize that reading a project is not easy. There can be multiple entry points and exits. Some files may exist for build tooling, testing, or configuration and may not contain information that is immediately useful when trying to understand how the project works.

As a beginner in open source, it is important to know how to read and understand a project before contributing. This helps you build a mental model of the codebase, understand how components interact, and identify which files need to be modified when adding a feature or fixing a bug.

Let's start with the basics.

This time we are going to read a Python package. Python projects are generally easier to navigate because they often have a straightforward and descriptive file structure.

When opening a Python repository, the first thing I look for is the root-level files:

project/
├── README.md
├── pyproject.toml
├── requirements.txt
├── LICENSE
├── tests/
└── some_lib/

The README is often the best place to start. It usually provides an overview of the problem the project is trying to solve, how the library or application is intended to be used, and the main features it offers.

Most maintainers include usage examples, and these examples often reveal the public API of the package and the common workflows users are expected to follow.

In many projects, you may also find additional documentation such as contribution guides, development setup instructions, or architecture documents. Spending a few minutes reading these files can provide valuable context and make the source code much easier to understand before diving into the implementation.

Find the Package Directory

The package directory usually has the same name as the library:

requests/
numpy/
fastapi/
your_package/

This folder contains the actual source code.
If you don't see any folder like these and there is __init__.py file in root directory that mean you are already in that folder

Open `init.py`

One of the most useful files for understanding a Python package is __init__.py.
For example:

from .client import Client
from .models import User
from .connection import InternalConnection

__all__ = ["Client", "User"]

This file often acts as the package's public interface. It tells us which classes and functions the author expects users to interact with.
From this single file we can already discover important modules:

client.py
models.py __all__ keyword is a list that tells Python which names should be exported when someone uses:

from package import *

In this example it will export only Client and User it does not import any internal functions.
Without __all__, Python imports all names that don't start with _. A common misconception is that __all__ makes things private. It does not. Even if it is exporting only Client and User you can still access some other class or function like InternalConnection

When reading real libraries, think of __all__ as the package author saying:

"These are the names I officially support. Everything else is internal implementation detail."

Follow the Public API

Instead of reading every file, start with the objects exposed in __init__.py.
For example, if you see:

from .client import Client

go to client.py and inspect the Client class.

Look for:

Constructor (__init__)
Public methods
External dependencies
Internal modules it imports

This often leads you through the project's main execution path.
Sometimes you will see * in __init__ or any function args. If you know python well you will understand what that mean but as this covers beginner let me explain.
There are 3 types in which you can find * used:

using only * like this example:

def greet(name, *, age):
    print(name, age)

In this case we can call this function as greet("Priyanshu", age=18) but not like this greet("Priyanshu", age=18) what that mean * by itself marks that args after that will start as keyword-only arguments. This makes code a lot more readable and bug free.
That is why when we use some big libraries we use like this.

In this case you will find it like

def func(*args):

Here * means collect extra positional arguments into a tuple. This is used where function don't know how may arguments can be passed so using *args it collects all into a tuple. You can find it used like this too

nums = [1, 2, 3]  
add(*nums) # calls like add(1, 2, 3)

Here * spreads the list into tuple like args giving some convenience.

Last is like this example

def func(**kwargs):

In this case ** mean key value pair arguments that mean a dictionary can be passed in the way *arg is for tuple.
and it can be called similarly but with dictionary items spreading.

kwargs = {  
"name": "Priyanshu",  
"age": 18  
}
print(**kwargs)
# or like this 
print(name="Priyanshu", age=18)

both ways are valid and common to use.

Following Imports

After understanding the public API, the next thing I usually do is follow imports.
Almost every file imports something from somewhere else. These imports are like roads connecting different parts of the project. If you can follow these roads, you can slowly understand how the whole system works.

For example, suppose you open client.py and see something like this:

from .database import Database
from .embeddings import Embedder

This immediately tells us that the Client class probably depends on a Database and an Embedder. We don't know exactly what they do yet, but now we have some direction. Instead of reading random files, we can follow these imports and see where they lead.

As you continue doing this, a picture starts forming in your head. You begin to understand which files are responsible for storing data, which files handle business logic, and which files simply provide utility functions.

One thing I learned while reading open-source projects is that codebases are not just collections of files. They are graphs. Every import creates a connection between two parts of the system. The more connections you follow, the clearer the architecture becomes.

Understanding Object Creation

At some point you will encounter a class that creates other objects inside its constructor.

For example:

class Client:
    def __init__(self):
        self.db = Database()

When I see code like this, I immediately open database.py.

The reason is simple. The Client class is telling us that it depends on a Database object. If we want to understand how the Client works, we also need to understand what the Database is doing.

This process repeats again and again. One file leads to another file, which leads to another file. Gradually you start discovering the major components of the project and how they communicate with each other.

Finding Where Execution Starts

A common question beginners have is: "Where does the project actually start?"
The answer depends on the project, but every application has an entry point somewhere.
Sometimes it is a main.py file. Sometimes it is a CLI command. Sometimes it is a web server startup script.

You may come across something like this:

def main():
    app.run()

if __name__ == "__main__":
    main()

This is often a good place to begin tracing execution because it shows what happens when the program starts.

Libraries are slightly different. Instead of looking for a main function, you usually start from the public API exposed through __init__.py and follow the path from there.

Following a Single Feature

When I first started reading open-source projects, I tried to understand everything at once. That approach never worked.
A better approach is to pick a single feature and follow it from beginning to end.
Suppose the package exposes a search function:

client.search("python")

Instead of reading every file in the repository, focus on tracing a single feature from start to finish.

In this example, start by locating the implementation of the search method and follow its execution flow. As you move through the code, observe which functions are called, how data moves between components, whether a database is involved, if embeddings are generated, and how the results are processed before being returned.

Following a single execution path naturally reveals the most important parts of the project and how they work together. This approach is much easier to manage than trying to understand the entire codebase at once.

Reading Configuration Files

Once I have a rough understanding of the source code, I usually go back and look at configuration files.
One of the most useful files in modern Python projects is pyproject.toml.
At first glance it may look like a boring configuration file, but it often contains a lot of useful information.
It can tell you the package name, version, dependencies, build system, and sometimes even command line entry points.
If you see dependencies such as FastAPI, SQLAlchemy, or Pydantic, you can already make educated guesses about the architecture of the project before reading more code.

These clues become surprisingly useful when navigating larger repositories.

Do Not Ignore Tests

For a long time I completely ignored test files because I thought they were only useful for maintainers.
That was a mistake.
Tests are often some of the easiest files to read in a project because they show how the author expects the library to be used.
A complex implementation might take hundreds of lines to understand, but a test can often explain the same behavior in ten lines.
Many times I have understood a feature faster by reading its tests than by reading the implementation itself.

So whenever you feel lost, open the test directory and see how the library is being used there.

Using Your Editor as a Navigation Tool

Modern editors make reading code significantly easier.

When you see a function call, use "Go to Definition". When you see a class name, jump to its implementation. When you want to know where something is used, use "Find References".
These features save an enormous amount of time because you do not have to manually search through dozens of files.

Over time, navigating a codebase becomes less about scrolling through folders and more about jumping directly between related pieces of code.

Final Thoughts

Reading code is a skill that improves through repetition.
The first few projects you read will probably feel confusing. There will be unfamiliar files, unfamiliar patterns, and many moments where you have no idea what is happening.

That is completely normal.

The goal is not to understand every line of code. The goal is to build enough context to understand how the project works, how data moves through the system, and which files are responsible for a particular feature.

Once you reach that point, the project starts feeling much smaller. Instead of seeing hundreds of unrelated files, you begin to see a set of connected components working together to solve a problem.

Reading code also makes you a much better developer. Every project teaches you something new. Sometimes you discover a cleaner way to structure code. Sometimes you learn a pattern or technique you had never seen before. Other times you may realize that a certain implementation can be improved and start thinking about alternative approaches yourself.

This is one of the biggest benefits of reading open-source projects. You are not only learning how a specific project works, but also how experienced developers think, organize code, and solve problems.

And that is usually the point where contributing stops feeling intimidating and starts becoming enjoyable. Instead of just consuming code, you start learning from it, questioning it, and eventually improving it.

AND remember this is Priyanshu Verma

Why I’m Shifting From TypeScript to Rust This Year

Priyanshu Verma — Thu, 12 Feb 2026 15:57:59 +0000

This year I made a decision that feels uncomfortable but necessary. I’m shifting my primary focus from TypeScript to Rust.

This is not a dramatic rage-quit. It’s not “TypeScript is bad.” It’s not hype chasing. I still respect TypeScript. I still use it. The browser understands JavaScript and nothing else, so obviously TypeScript will always be part of modern development. But relying on it as my core long-term skill no longer feels right.

This shift is about depth. It’s about leverage. It’s about choosing friction now to gain strength later.

For the last few years, TypeScript has been comfortable. Extremely productive. You can build full-stack apps quickly. The ecosystem is massive. Documentation is everywhere. You hit a problem, search it, and there are already ten answers. AI tools autocomplete entire features in seconds. It feels like flying.

But that comfort started bothering me.

When something becomes that smooth, that frictionless, you have to ask yourself what is actually defensible about your skill.

The reality is this: AI models are exceptionally good at JavaScript and TypeScript. The training data is enormous. Patterns are repetitive. The runtime is forgiving. Even senior-level TypeScript code can be generated in seconds. That doesn’t make TypeScript useless. It just means the barrier to output is extremely low, for both humans and machines.

And when a skill becomes low-friction for machines, its long-term defensibility changes.

This is the uncomfortable part people avoid talking about.

If you are only building in a language that AI understands better than you do, you are competing with a tool that doesn’t get tired and doesn’t need context switches. That’s not a great long-term strategy.

So I started thinking differently.

What if instead of optimizing for speed of building, I optimize for depth of understanding?

That’s where Rust entered seriously.

Rust is not just another backend framework. It’s not a “Node alternative.” It’s a systems language. That distinction matters. With Rust you can build web servers, sure. But you can also build operating systems. Game engines. Networking stacks. Embedded firmware. Database engines. Compilers. You can compile to Linux, Windows, macOS, ARM, and even bare-metal. You can run without an operating system at all.

That changes how you think.

You are no longer just writing API routes. You are dealing with memory layout, ownership, concurrency models, lifetimes, deterministic behavior. You are closer to the machine. That proximity forces you to understand what is actually happening.

TypeScript is about developer experience. Rust is about system integrity.

And I’m starting to prefer system integrity.

One of the biggest reasons is how Rust handles correctness. The Rust compiler is relentless. People joke about the borrow checker, but it’s not just a technical feature. It’s a philosophy. The compiler does not trust you. Even if you are confident your program will work, Rust will question every edge case.

And this reminds me of something very simple.

The Rust compiler is like your mom.

Imagine you say, “I want to go on a road trip.”

You are confident. You have planned everything. You know how to drive. You are excited.

And your mom says, “What if the car crashes?”

You respond, “That won’t happen.”

She says, “What if it does?”

You say, “That’s hypothetical.”

She says, “Still. What if?”

That is Rust.

Even when you are sure everything is fine, Rust forces you to think about the hypothetical crash. What if this reference outlives its owner? What if this task shuts down mid-execution? What if this error isn’t handled? What if this memory is accessed concurrently? What if the network drops? What if the process exits early?

It doesn’t accept optimism. It demands preparation.

And that’s exhausting at first.

Rust is not dopamine-friendly. It does not give you the instant satisfaction that scripting languages do. You don’t just write code and watch it magically run. You fight the compiler. You refactor. You restructure. You rethink ownership. You redesign your data flow. Sometimes you feel like you are losing to a machine.

But something changes after a while.

You realize the compiler is not fighting you. It’s training you.

It forces you to think in systems. It forces you to think about failure before failure happens. It forces you to design intentionally. That kind of discipline is uncomfortable but powerful.

TypeScript gives you fast iteration. Rust gives you strong guarantees.

TypeScript gives you flexibility at runtime. Rust gives you certainty at compile time.

TypeScript gives you speed of shipping. Rust gives you durability of shipping.

And I started asking myself what I want more of.

Another major factor is deployment philosophy. I’ve grown tired of runtime stacks layered on runtime stacks. Node behind reverse proxies inside containers inside cloud abstractions. Hundreds of megabytes deployed for something that fundamentally just responds to HTTP requests.

With Rust, you can compile a backend and embed a compiled frontend into a single binary. No Node runtime. No reverse proxy required. No external dependencies. Copy the binary to a Raspberry Pi and run it. It works. It doesn’t ask what version of something is installed. It doesn’t need npm install. It doesn’t need Docker if you don’t want it.

That simplicity is addictive.

It feels honest.

You build something. You compile it. You ship the artifact. That artifact is the product. There is something clean about that model. One more thing is that it is a binary even if someone gets it. they cannot see the code that we can say is a security feature.

Now, why not Go?

Go is good. I respect it. It compiles fast. It has strong concurrency primitives. It’s pragmatic. But for me, personally, it never felt expressive enough. The syntax feels procedural in a way that doesn’t match how I think. I come from an object-oriented background, and while Rust isn’t traditional OOP, its type system feels far more expressive and powerful.

More importantly, Go isn’t a systems language in the same sense. You cannot compile Go to true bare-metal targets the way you can with Rust. Rust’s reach is broader. That matters to me. I don’t want to just write web servers for the rest of my life. I want the option to go deeper.

This isn’t about language wars. It’s about surface area of capability.

Rust expands mine.

There’s also timing. Rust is modern but still relatively young in mainstream adoption. The market is growing. The ecosystem is maturing. But it’s not saturated. Becoming deeply competent in Rust today feels like investing early in something foundational.

And beyond market thinking, it feels intellectually honest. Rust makes you slower in the beginning. It removes shortcuts. It punishes careless assumptions. It makes you confront complexity directly instead of hiding it under runtime flexibility.

It is not beginner friendly. It is not easy. And that’s precisely why it’s valuable.

If someone wants instant feedback, fast feature velocity, and quick wins, Rust will feel like a nightmare. The compiler errors can be brutal. You might write code that logically makes sense, and Rust will reject it for lifetime reasons you didn’t even consider. It feels unfair at first.

But over time, you realize it’s making you sharper.

You start anticipating ownership issues before writing code. You design APIs differently. You think about shutdown signals. You think about memory footprint. You think about concurrency boundaries. You think about determinism.

You stop assuming things will work. You prove that they will.

That’s a different mindset.

I am not abandoning TypeScript. That would be unrealistic. The browser speaks JavaScript. Frontend ecosystems are deeply tied to it. Even if you use WebAssembly, JavaScript is still there. So TypeScript will remain part of my workflow.

But I don’t want it to be my only layer of competence.

Relying solely on TypeScript feels narrow. Rust feels expansive.

This shift is not about rejecting comfort for the sake of pain. It’s about choosing growth over convenience. I know this year will be slower in terms of output. I know I will spend more time reading compiler errors than shipping features. But I also know that the understanding I build will compound.

In a world where AI can generate boilerplate instantly, the real advantage is not how fast you can scaffold an app. It’s how deeply you understand the systems you are building on.

Rust pushes you toward that depth whether you like it or not.

And maybe that’s what I need right now.

Not speed. Not comfort. Not instant validation.

Depth.

That’s why I’m shifting.

Homelabing - Local to online setup suppressing CGNAT

Priyanshu Verma — Thu, 25 Dec 2025 12:44:45 +0000

In the journey of creating a Hub before we start programming or building website lets setup Pi in a way where we can access it on mobile internet.
For that we can use Cloudflare Tunnel or Tail Scale, but the problem is both are overpowered for this setup I don't want to use Cloudflare as an intercept between Me and my data. I can't trust any 3rd Party. So, not going to use that. I will not use Tail Scale because some features are awesome. But, it to use it I need to make accounts and every new person joins the network need to create an account on Tail Scale and I don't think. I will create an account on 3rd party to access my self-hosted server.
Now, This leads to setup my own VPN on Pi and connect to devices when I want to connect with Mobile network.

For That, I will be going to use WireGuard.
Why? Because it is open-source light and free.
Let's set it up.

Installing WireGuard on Pi.

sudo apt update
sudo apt install wireguard -y

And verify by printing it's version

wg --version

Now, for this to work, we need to create Pi's Private and Public Keys. To create, follow this.

sudo mkdir -p /etc/wireguard
sudo chmod 700 /etc/wireguard
sudo wg genkey | sudo tee /etc/wireguard/server_private.key | sudo wg pubkey | sudo tee /etc/wireguard/server_public.key

Now, you will find,

-rw------- 1 root root 45 server_private.key
-rw------- 1 root root 45 server_public.key

inside /etc/wireguard if not, then regenerate the keys.

For me, it worked. Now let's add some config.

sudo nano /etc/wireguard/wg0.conf

This config

[Interface]
Address = 10.10.0.1/24
ListenPort = 51820
PrivateKey = <PASTE server_private.key CONTENT>
SaveConfig = true

I am using 10.10.0.1/24 for subnet, as It is easy to remember for me and will not conflict with anything.
Now I will enable IP forwarding
by adding net.ipv4.ip_forward=1 inside /etc/sysctl.conf and apply the changes with.

sudo sysctl -p

It should output 1
Now, let's enable wireguard

sudo systemctl enable wg-quick@wg0
sudo systemctl start wg-quick@wg0

and verify with sudo wg

Now it is time to install WireGuard Official Android app on mobile and add it as a Peer.
Will create a new tunnel generate public and private keys then on Pi in /etc/wireguard/wg0.conf will add peer in bottom.

[Peer]
PublicKey = <PHONE_PUBLIC_KEY>
AllowedIPs = 10.10.0.2/32

and reload wg

sudo wg-quick down wg0
sudo wg-quick up wg0

finally on mobile WireGuard will add

Address = 10.10.0.2/32
DNS = 192.168.1.1 # your gateway IP

and add a Peer over there

PublicKey = <SERVER_PUBLIC_KEY>
Endpoint = <YOUR_PUBLIC_IP>:51820
AllowedIPs = 0.0.0.0/0
PersistentKeepalive = 25

Now at this point I don't have a static Public IP as my ISP don't know me doing homelabing. If you are able to get static IP then use it otherwise there is one workaround.
I will use DuckDNS, this will help me to mimic my dynamic IP to look like static and on top of it. It is free, Lightweight and No vendor lock-in also it is not a proxy but a DNS record. To hook it go to https://www.duckdns.org
create an account and add a domain name it whatever you want, it is not a public address just a link to your public IP.
and use that subdomain as an endpoint.

We need to make a cron job to update the latest IP on DDNS. For that on Pi will create a Script to do that.

sudo apt install curl -y
mkdir -p ~/duckdns
nano ~/duckdns/update.sh

using this

#!/bin/bash
echo url="https://www.duckdns.org/update?domains={YOUR_SUBDOMAIN}&token={YOUR_TOKEN}&ip=" | curl -k -o ~/duckdns/duck.log -K -

Update YOUR_TOKEN from DDNS token and YOUR_SUBDOMAIN with your own subdomain.
Now give some permissions and test it.

chmod +x ~/duckdns/update.sh
./duckdns/update.sh
cat ~/duckdns/duck.log

It should give OK. NOT KO if it says KO then you are KO setup failed check your token and subdomain.
Now let's add a cron

crontab -e

and adding this to run every 5 minutes

*/5 * * * * ~/duckdns/update.sh >/dev/null 2>&1

Now, let's try to connect turn VPN on in mobile it should connect and by running

sudo wg

You will see a device connected saying
latest handshake: X seconds ago

If not then Hello my friend me too. Our ISP uses CGNAT (Carrier-Grade Network Address Translation) so they can control the router and as a user we cannot forward and port on IPv4.

Then, What to do??
You know what, after hours of research, I found that I cannot forward port by any mean, so the last option is to use Tail Scale, and so far one of the requirements broke. But, it is not my mistake, I cannot control my ISB behavior so. I have to use Tail Scale.

So, we don't need DuckDNS now let's remove the cron and wireguard.
If you followed along, then you will feel what wasting time looks like.

crontab -l
rm -rf ~/duckdns
sudo systemctl stop wg-quick@wg0
sudo systemctl disable wg-quick@wg0
sudo apt purge wireguard wireguard-tools -y
sudo apt autoremove -y
sudo rm -rf /etc/wireguard

Then remove that port forward line from sudo nano /etc/sysctl.conf
Now, create an account at https://login.tailscale.com
Then add the devices and install it on Pi,

curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up

and done online.
You can check it on tailscale dashboard.

Now, we can focus on building online Pi is on local and online.
Just run a server running and good to go.

Homelabing interesting enough to say but why

Priyanshu Verma — Thu, 25 Dec 2025 12:06:15 +0000

So far, I tried to not tie myself to big techs for some basic stuff. Like, from starting I switched products for password management, media, IOT management. But stuck with vendor lock and tied to subscriptions and limited control.
Finally, decided to make a home server.
For me, it is not like buying a big rag, Plenty of storage, RAM, and heavy suite like proxmox or other. I started very small from a Raspberry Zero 2W. Why, because ~~money~~ I am a professional developer, I know to build a system that works in limited resources. This is not a bug but a feature. I will try to build tech from scratch that I can run on it. For Media and other IOT stuff.
Also, This is only for my use, I am not building a robust system for millions of users using simultaneously, It is for me only or my family.

This homelabing series is going to be technical and not for production systems. I don't care if I fry my Pi, it is just a learning journey, I might upgrade hardware in the future. We can start with Pi only.

For more context let me tell you what is my ground for now.

Raspberry Pi Zero 2W Board (ORIGNAL)
- 1GHz quad-core 64-bit Arm Cortex-A53 CPU
- 512MB SDRAM
- 2.4GHz 802.11 b/g/n wireless LAN
- Bluetooth 4.2, Bluetooth Low Energy (BLE), onboard antenna
- Mini HDMI® port and micro USB On-The-Go (OTG) port
- microSD card slot
- HAT-compatible 40-pin header footprint (unpopulated)
- Micro USB power
SAMSUNG POWER Prime 256 GB Micro SDXC Class 10 Memory Card
- capacity: 256 GB
- Read Speed: 160 MB/s
Power Adapter normal 2A for Powering Pi

That's the ground I have installed Ubuntu 64bit lite on memory card and setup Pi with a static IP from home router using it's Mac Address.
So far, nothing is installed, and I can ssh into Pi from my PC that is the bare minimum. Also, One more thing is There are two partitions I have divided unusable ~235GB of storage in ~200GB + rest. So I can deploy stuff on rest part which is ~35GB and can keep the media and all other stuff in that ~200GB part. Safe and simple.
Also, I have installed Samba and did setup so I can mount big 200GB part to my Windows PC and store files. It is easy no big deal.

This is the final ground from where I will be making it The way I want, which includes:

Simple IOT controlling Dashboard for remotely switching lights at home and other fun stuff.
host simple images and videos downloaded and store, no processing or indexing simple static serving.
Simple authentication so, I can restrict certain things and add my family members to access media.
Do it in a way so I can add more stuff in future like internal messaging or calling. These are basic enough right, remember we will be doing it under usable 416M ram.

I do have some rules that I will follow. I don't want to mess things by using software that I feel is not giving me control and heavy on the system.

NO cloud platform for processing or intercepting traffic like Cloudflare tunnel
No REACT, for building frontend.
Keep it accessible on remote location, (aka open to use even I am not connected to home network).
less latency
Easy to use for non-techy people.
SECURED!

Sounds crazy, but it is doable.
Let's see how it goes…

You Want a Future in Tech? Stop Building Templates.

Priyanshu Verma — Tue, 25 Nov 2025 16:33:30 +0000

Hi
So, Now everything around us has some sort of AI badge. Even on my Washing Machine. Seriously, Now it recommends me to select the mode based on my route.
Let me give you a brief introduction about me. I am a developer who works major in Web. And with AI now, I am able to make beautiful UIs in seconds and write code build my side projects in Hours. I just need a prompt and some <TAB> <TAB>.
Done.

Why I wrote this article

If we see it from managing or productive side then it is awesome. You are now able to do Hours of work in minutes. If you are in cooperate or have a job then It fits in very awesome way.
But, if it is easy then anyone can do it.
I observed that WEB DEVELOPMENT is not a hard thing anymore any one that know what to build can make websites.
Well. Now you say building UI with AI is nothing main thing is managing it, sustaining it. I know that. I am not saying any non tech can build or sustain it. I am talking about the competition. Already the competition in Web Development is very high very other person is build an AI SaaS for nothing.
And I think, Now companies and Startups don't need a lot of people for Web Development and which means less Jobs and less opportunities as most of the work is simple.

As a student or person who want to get in tech or want to learn something to earn money in long term. The path of learning about Web Development not looks safe anymore.

What is the future of Web development

If we see the growth of AI and the tools that are available right now like we have IDEs AI integrated and some generic packages or libraries which works very fine with AI like if We need the world of web and intelligence if AI we can conclude at using tech likeshadcn/ui and Better Auth for authentication also for backend serverless functions, ORMs like Prisma helps to build any product very easily Everything is open and works fine with AI. Famous AI models knows about these technologies and they work very effectively on these.

And in future it will get more better because almost 90% of the code of any website is open and there is a lot of data on which AI models can be trained. Every open-source frameworks, public tutorials, GitHub repos, countless blog posts, StackOverflow questions, boilerplates, templates, and design systems are public by default. And the feedback loop of the web makes things even easier for AI. If you write HTML or React, you instantly see the result. If something breaks, the browser reports the exact line.

AI eating this domain of tech very rapidly.

What do learn then

Before answering this question I want to ask why you get in tech. The problem with today's generation is that they think I will get in tech build websites and earn money. You and I we both know that the most easiest job in tech is to build websites and people choose it because it is simple.
The harsh truth is people became lazy and they want instant feedbacks that's why nowadays we don't see any projects like Linux, Git or some revolutionary tech because they are depended on AI. and AI knows Web better so defaults to it.

So, What to do uh? what should i learn. The answer is very simple do what AI cannot do. Programming is not dead. The humours are faking it. AI is not able to build a native app for windows, android, It is not able to build custom software for specific hardware, it cannot do cyber security. Literary it fails to code in system languages like C, C++, or Rust, Even in kotlin. You know why because the information build about these is less and building a solution for hardware or building an native app or doing cyber security stuff needs on demand actions that AI cannot replicate. Fields like DevOps AI cannot replace as it needs to work on demand according to external stimuli.

What is the point of building native

So, my friend why you should build a mobile app with native tech you can use JavaScript to build it.
Technically it works. It shows a screen, buttons click, cool. But then I’m like… bro look at our devices. We literally have super powerful chips now, like phones with GPUs that are better than old consoles, laptops with crazy CPUs, RAM is cheap, SSD is fast… and still I open 20 chrome tabs and the whole thing starts dying. I run a game on my RTX and it can’t even stay on 60 FPS. Why? Why are games 150GB now? Why is a simple chat app using 700MB RAM?

And the sad part is we all kinda became lazy. We just accept it. No one wants to build their own game engine because “eh too much work.” No one wants to create their own UI library because “just npm install something.” Everything is off-the-shelf now. Zero originality. Every website looks copy-paste same. Every game feels same. Every app has that same template vibe. We’re not optimizing, we’re just piling more stuff on top.

Can AI really Replace you

See, It depends on your skills.
Honestly… depends on what you’re doing. Like, seriously. If your whole skillset is just copying templates, building basic CRUD, making landing pages, doing the same loops everyone is doing, then yeah AI is on your heels. Because AI is literally trained on that stuff. It has seen millions of those patterns.

But the funny thing is the stuff I talked about above the messy parts, AI can’t touch that. It just can’t. It doesn’t know how to handle real hardware problems, memory issues, native bugs, performance tuning, cybersecurity attacks, networking glitches, weird OS behavior. That stuff is too real. Too unpredictable. Too out of the training data bubble. Only humans can deal with it.

And this whole “recession because of AI” panic? Bro, half of it is just people sitting in their comfort zone. They open VS Code, type two prompts into an AI agent, feel like a CEO for a minute, and then scroll reels for half an hour. Then they wonder why they’re not growing and blame AI like it’s hunting them personally. It’s not AI. It’s the fact that we all got a bit too comfortable building comfort zone software.

AI is not replacing the people who actually touch the real stuff the complex systems, the deep logic, the broken pipelines, the debugging nightmares, the places where you can’t just hallucinate a solution. AI is replacing the lazy, repetitive parts, the factory style coding that didn’t need a human in the first place.

Conclusion

At the end of the day it’s actually not that deep.
You wanna stay irreplaceable?
Just stop doing the bare minimum and calling it a “skill.”
Get out of the comfort zone stuff.
Start touching the parts of tech where AI straight up falls on its face.
Systems, hardware, native, security, actual optimisation is the real engineering side of things.

That’s literally the whole game.

Self-Hosting a Password Manager on a Pi Zero 2W

Priyanshu Verma — Sun, 28 Sep 2025 06:37:37 +0000

As a developer and tinkerer, I love taking control of my digital environment. This weekend, I decided to host my own password manager. My goal was simple: keep all my sensitive data private, accessible from multiple devices, while learning more about self-hosting, TLS, reverse proxies, and Docker on a low-power device. Here’s how the journey unfolded, along with problems I faced and solutions I discovered.

Choosing the Right Tool

I started by exploring the options. Cloud-based managers like 1Password or Bitwarden are convenient, but I wanted full control. Among self-hosted options, Vaultwarden stood out. It’s a lightweight, unofficial Bitwarden implementation, supports TOTP, and can run on low-resource devices like a Pi Zero 2W.

For a solo developer like me, Vaultwarden was ideal: small footprint, actively maintained, and feature-rich.

Step 1: Preparing the Pi

I installed Ubuntu on the Pi Zero 2W and set up SSH access to manage it remotely. Since I planned to use Docker, I made sure the system was updated and installed docker and docker-compose.

After verifying Docker installation with docker --version, I was ready to pull Vaultwarden.

Step 2: Running Vaultwarden in Docker

My first attempt to run Vaultwarden showed an important warning:

No persistent volume!

This meant that if the container stopped or was updated, all data would be lost. Lesson one: always map persistent volumes. I created a vw-data directory and updated docker-compose.yml:

vaultwarden:
  image: vaultwarden/server:latest
  container_name: vaultwarden
  restart: unless-stopped
  environment:
    ADMIN_TOKEN: "changeme"
    ROCKET_WORKERS: 1
    ROCKET_PROXY_PATH: "/pass"
  volumes:
    - ./vw-data:/data
  expose:
    - "80"

The ROCKET_PROXY_PATH="/pass" was critical.

Step 3: Subpath vs Subdomain

At this stage, I faced a design choice:

Option 1: Subdomain (vault.mydomain.com)
Option 2: Subpath (mydomain.com/pass)

Initially, subdomains seemed neat and clean. But there are trade-offs for self-hosted services:

Certificate complexity: With a subdomain, I would need a separate certificate for vault.mydomain.com. With a local or internal CA, managing multiple subdomains could get messy.
Future services: I wanted to host other tools (like a personal wiki or file server). Using a subdomain for every service could clutter DNS and complicate VPN/local routing.
Tailscale & mobile access: VPN solutions sometimes resolve only the main hostname. Subdomains occasionally fail or require additional DNS tweaks.

A subpath approach (/pass) solved these issues:

One certificate covers the main domain.
Reverse proxy handles routing internally to different services.
Makes scaling to multiple services simpler without touching DNS or VPN settings.

Hence, I chose /pass for Vaultwarden and left / free for other services. This required careful reverse proxy configuration, which I’ll describe next.

Step 4: Reverse Proxy with Caddy

I used Caddy as a reverse proxy to handle HTTPS and route /pass to Vaultwarden. My first attempt was basic:

panda.local {
    tls internal
    encode zstd gzip
    reverse_proxy vaultwarden:80
}

Problems immediately appeared: assets broke, WebSockets failed, and trailing slashes caused the interface to misbehave. Vaultwarden expects requests at the root, so a simple proxy was insufficient.

After research and testing, I implemented:

handle_path /pass/* {
    reverse_proxy vaultwarden:80 {
        header_up X-Real-IP {remote_host}
        header_up X-Forwarded-Proto {scheme}
        header_up X-Forwarded-Host {host}
        header_up X-Forwarded-For {remote_host}
    }
}

handle_path /pass/* strips the /pass prefix before forwarding requests to Vaultwarden. This solved:

Asset loading issues
WebSocket failures
Trailing slash inconsistencies

Step 5: TLS and Certificates

Since I wanted both desktop and mobile apps to trust my self-signed setup, I used mkcert to generate a local CA and certificates:

mkcert panda.local
mkcert pass.local

Copying the root CA to my desktop required careful path handling due to spaces in usernames, and mobile installation required converting .pem to .crt and manually trusting the certificate. Once done, browsers and apps stopped showing SSL warnings.

Lesson learned: TLS is critical, but self-hosting means you need to manage trust explicitly on each device.

Step 6: Remote Access with VPN

I didn’t want to open my home network to the internet. Using a VPN-like solution, I could access Vaultwarden remotely without exposing ports. Initially, there were DNS resolution issues with subdomains, but after adding entries to local hosts and configuring certificates correctly, remote access worked smoothly.

This setup also allowed me to think ahead: by using /pass, I left the root path free for future services, creating a small personal service ecosystem.

Step 7: Troubleshooting Common Issues

Here’s a list of the main problems I faced and how I solved them:

No persistent volume: Fixed by mapping ./vw-data:/data in Docker Compose.
Subpath asset/WebSocket issues: Resolved with handle_path /pass/* in Caddy.
SSL warnings: Generated local CA with mkcert, imported into desktop and mobile devices.
Trailing slash problems: Also handled by handle_path.
Remote access quirks: Fixed via local DNS entries and correct certificate paths.
Subdomain vs subpath confusion: Choosing subpath simplified certificates and routing for multiple services.

Step 8: Final Setup

By the end of the day, I had:

Vaultwarden running under /pass on the Pi Zero 2W
Full HTTPS support trusted on desktop and mobile
Remote access without exposing home network ports
Persistent storage for all passwords

Testing everything — creating accounts, logging in from mobile apps, syncing TOTP codes — everything worked flawlessly.

Reflections and Lessons Learned

Self-hosting is rewarding, but it comes with challenges:

Docker volumes prevent accidental data loss.
Reverse proxies can make or break subpath hosting.
TLS and certificates are critical; trust must be manually managed for internal or mobile clients.
Subpath routing simplifies multi-service hosting, while subdomains require additional DNS and cert management.

Small fixes like handle_path or importing a CA may seem trivial, but they make the difference between a broken interface and a smooth user experience.

Conclusion

Self-hosting Vaultwarden on a Pi Zero 2W might seem overkill for a single user, but for me, it was about control, privacy, and learning. The Pi handled everything surprisingly well, and Vaultwarden proved to be a solid, lightweight solution.

For developers looking to take control of their digital lives: start small, expect roadblocks, and treat each error as a learning opportunity. In the end, there’s immense satisfaction in seeing your own system, from certificates to WebSockets, working exactly the way you designed it.

Ever wondered why Windows or Android updates are so huge? This isn’t just about code – it’s about an entire hidden world of engineering farms, power consumption, and logistics. My new article opens that black box.

Priyanshu Verma — Sun, 17 Aug 2025 10:47:14 +0000

Priyanshu Verma

Aug 17 '25

The Hidden Engineering Behind Building Massive Software Images

#programming #cloud #architecture #learning

7 min read

The Hidden Engineering Behind Building Massive Software Images

Priyanshu Verma — Sun, 17 Aug 2025 10:46:09 +0000

Hi,
You guys know me I try doing something new always so, recently When I first stared at the AOSP tree and saw the download size, I thought someone was pranking me. Hundreds of gigabytes? Thousands of repos? How could anyone reasonably build that on a laptop, even if it had more cores than a nuclear reactor? The truth pulled me down a rabbit hole I didn’t expect — one that rewrites what “writing software” even means.

Let me take you on a short film of how a piece of code becomes a massive, signed, tested, shippable image — whether that image is your phone’s system update, a Windows release, or a billion-dollar AAA game. This isn’t just about compilers and make; it’s about orchestration, scale, people and machines working together in a choreography most users never see.

I want you to imagine two scenes.

Scene one: a lone engineer, late at night, running make on a workstation. The machine hums. The screen scrolls error messages and warnings. The coffee goes cold. This image is still real in small projects, but it is increasingly a nostalgic relic.

Scene two: a humming build farm, rack after rack of machines, jobs queued and parallelized across continents, caches consulted in milliseconds, thousands of tests flaring green or red. Keys and signatures are applied in a secure enclave. An image rolls down the pipeline and — eventually — reaches a device on a user’s desk via an OTA update. That is the new reality.

Why the chasm between these two scenes? Why can’t we just “compile the code” and be done?

It starts with scale. Modern operating systems and flagship apps aren’t a few thousand files stitched together. They’re an orchestra of components: kernels, drivers, vendor binaries, runtime engines, media libraries, UIs, services, apps, localization packs, and every test harness you can imagine. Where a small embedded project might be measured in thousands of lines, Windows and Android live in the tens-of-millions-of-lines regime. Chrome, Android, and game engines bring along enormous native code, Java or managed runtimes, and gigantic asset pipelines. Each of these pieces has its own build steps, expectations, and quirks.

But scale alone doesn’t explain the orchestration required. Building a modern image is not just compiling source; it’s a multi-stage process that touches toolchains, packaging systems, signing infrastructure, artifact storage, and certification systems. You don’t merely turn C into binaries; you:

run code generators and preprocessors,
produce architecture-specific binaries (x86_64, arm64, etc.),
link enormous binaries (link time optimization and profile-guided optimization can be highly sequential and heavy),
bundle resources (localization, images, precompiled shaders),
compress and sign artifacts with secure keys,
and then subject them to thousands of automated tests on real hardware.

All of that introduces latency and makes naive parallelism insufficient. Even on a machine with 64 cores, disk IO, linking, and serialized packaging steps become chokepoints. The first full build, the “clean build,” necessarily touches everything: read tens of thousands of files, compile, link, pack. That’s expensive in time and in energy.

Which brings me to a second, often-overlooked consequence: the environmental cost. When you multiply full builds by dozens of devs, nightly images, pull-request validations, and continuous fuzzing runs, the energy footprint becomes non-trivial. Companies running nightly images and large CI matrices are effectively operating mini data centers. This reality has pushed teams to optimize for two things: reduce redundant work and reduce waste. Caching (ccache, remote action caches), incremental builds, and smarter CI orchestration are not just developer conveniences — they’re carbon optimizations.

So, how do teams cope? How does any sane engineering organization ship anything at all?

They split the problem. Software at scale is modularized and specialized. Teams no longer touch the whole tree. An engineer working on the window manager builds the window manager. A kernel engineer builds the kernel. These are unit-of-work boundaries. The full single-piece build is delegated to a build farm — a distributed system of machines that can run the complete set of compilation tasks in parallel, honor dependency graphs, and cache results aggressively.

Distributed build systems are at the heart of this approach. Google’s Goma (migrated to Reclient in 2025) and remote execution, Bazel’s remote execution, Microsoft’s BuildXL, and similar technologies allow compilation tasks to be farmed out, cached, and reused. Instead of each developer redoing the same heavy work, cached object files are reused across jobs and builds. The result: individual iteration latency drops from hours to minutes, and the full heavy build happens in a controlled, centralized way.

But distributed builds are only part of the story. There’s also a surprising orchestration layer: artifact variants and packaging. One codebase can produce dozens — even hundreds — of artifacts. AOSP, for instance, doesn’t generate a single image; it generates system images for multiple device families, variants with different Google services or OEM customizations, boot images, vendor images, and more. Windows similarly produces many SKUs, localized images, and platform-specific packages. Games produce builds for multiple consoles, PC configurations, and packaged installers. Each variant multiplies the amount of work and the number of tests.

Testing is another dimension that explodes resource needs. It is rarely acceptable anymore to ship untested large changes. Integration suites, fuzzers, UI tests, driver compatibility tests — these run on thousands of device instances, sometimes physically — and failures require reproducing on identical hardware. So the engineering pipeline must not only build; it must distribute, schedule, and record results from a massively parallel test farm.

This reality created an entire discipline: build engineering and release engineering. These teams are the invisible logisticians of software. Their job is not product feature work; it’s to ensure developers can iterate quickly, builds are reproducible, artifacts are traceable and secure, and releases roll out smoothly. Reproducibility is a central obsession: you should be able to rebuild an artifact deterministically so you can trust what went out to users and trace bugs back to a specific build. That means pinning toolchains, hermetic builds, recorded inputs, and deterministic packaging — a lot of boring-but-important work.

To me, one of the most compelling stories is the contrast between eras. In the ’80s and ’90s, building an OS was something you could conceive on a single workstation. Engineers would go home and let overnight builds run. Today, building Windows or a AAA title is akin to running a rendering farm for a movie: assets measured in terabytes, precompiled shaders, baked global illumination, and platform certification pipelines. Modern AAA titles have asset pipelines that are as much a part of the build — with their own distributed render farms — as the code compilation itself. The product is rarely “code”; it’s code times assets times compatibility matrices.

Windows is a vivid example. Its codebase has accreted decades of code, multiple languages, legacy interfaces, and vast driver ecosystems. Microsoft’s approach is to carve the monolith into components. Engineers iterate locally on targeted subsystems and push changes. A centralized farm, armed with sophisticated orchestrators, then runs full builds and regression suites. For Windows, the human problem isn’t “write the code”; it’s “ensure that code plays nicely with tens of thousands of hardware permutations and years of backward compatibility.” Microsoft’s build infrastructure is built to manage that complexity — not to make building trivial, but to make shipping possible and predictable.

And if you’re wondering whether AI will change this equation — write code and let machines assemble everything — the answer is: AI will help, but it won’t replace the choreography. Writing code is only a shard of the whole. The pipeline, the reproducibility, the signing, the compatibility testing, the device certification, the incremental build caches, the keys and signing policies, the human coordination, the rollback strategies — these are socio-technical problems. AI can suggest changes, even generate modules, but someone still has to decide which artifacts to build, how they get tested, what gets signed, and how to remediate a failing rollout. In short, AI can be an assistant, not the orchestra conductor — at least not in the near term.

If you want to dig deeper, here are a few research and reading touchpoints I keep returning to while exploring this world:

Read the Bazel documentation and the philosophy of hermetic builds.
Explore Google’s papers and blog posts on Goma / remote execution (they explain why distributed compilation matters).
Look up BuildXL (Microsoft) and the evolution of Git Virtual File System (GVFS) for handling huge repositories.
Search for the “Reproducible Builds” project — it captures why deterministic builds are so central to security and trust.
Check out material on remote caching and action caches (how CI avoids doing repeated work).

I find the story irresistible because it reframes what building software means: it’s less about the solitary craft of typing functions and more about designing resilient industrial processes. The magic of a software update isn’t the few lines someone typed last week — it’s the invisible machinery that compiled, tested, signed, and safely delivered that update to millions without breaking everything else.

If you’re curious, here’s one practical way to feel this under your fingers: don’t attempt a whole AOSP build yet. Instead, pick a subsystem (a kernel module, a small Android service, a game asset pipeline task), build only that module locally, replace it into a prebuilt image, and iterate. You’ll experience first-hand the dramatic speedups that modular work affords, and you’ll see why entire teams focus on making those iterations as fast and reliable as possible.

When people ask me what the future of software engineering looks like, I don’t see a world where developers are obsolete. I see a world where “developer” means more — you’ll need to understand not only code, but caches, CI graphs, reproducibility guarantees, artifact signing, and the economics of compute. The boring parts — the build farms, the signing policies, the orchestration — are where trust is forged. And that, I think, is a story worth telling.

Thanks for reading to the end. I know you have time to waste then also checkout my Github

— Priyanshu Verma

DEV Community: Priyanshu Verma

Storing Documents: From a JSON String to Bytes on Disk

The Temptation of Raw JSON

Why BSON Changes the Game

The Twelve Bytes of an ObjectId

How Documents Live Inside Pages

The Anatomy of a Record

The Insertion Pipeline: From Shell to Disk

Reading Data: The Honest Linear Scan

Soft Deletion and the Concept of Tombstones

What We Built

Collections and the Catalog: How a Database Remembers What Exists

The Table of Contents for Your Database

The Anatomy of the Catalog Page

Null Terminators and C Strings

Why the Catalog Struct Is Completely Empty

Creating a Collection: The Journey of a Write

Listing Collections with a Cursor

Deleting a Collection: The Shift-Left Trick

Preventing Catastrophe with a Lock File

Where We Stand

The File Format: Pages, Headers, and Magic Bytes

Why Structure Matters

The Page: The Fundamental Unit

The File Header

What the Header Tracks

Binary Encoding

The Pager

Bootstrap: First Database Write

The File Layout So Far

Building a NoSQL Database in Rust, from Scratch: The Why

How to Read Open Source Code, Python Edition

Find the Package Directory

Open __init__.py

Follow the Public API

Following Imports

Understanding Object Creation

Finding Where Execution Starts

Following a Single Feature

Reading Configuration Files

Do Not Ignore Tests

Using Your Editor as a Navigation Tool

Final Thoughts

Why I’m Shifting From TypeScript to Rust This Year

Homelabing - Local to online setup suppressing CGNAT

Homelabing interesting enough to say but why

You Want a Future in Tech? Stop Building Templates.

Why I wrote this article

What is the future of Web development

What do learn then

What is the point of building native

Can AI really Replace you

Conclusion

Self-Hosting a Password Manager on a Pi Zero 2W

Choosing the Right Tool

Step 1: Preparing the Pi

Step 2: Running Vaultwarden in Docker

Step 3: Subpath vs Subdomain

Step 4: Reverse Proxy with Caddy

Step 5: TLS and Certificates

Step 6: Remote Access with VPN

Step 7: Troubleshooting Common Issues

Step 8: Final Setup

Reflections and Lessons Learned

Conclusion

Ever wondered why Windows or Android updates are so huge? This isn’t just about code – it’s about an entire hidden world of engineering farms, power consumption, and logistics. My new article opens that black box.

The Hidden Engineering Behind Building Massive Software Images

The Hidden Engineering Behind Building Massive Software Images

Open `init.py`