DEV Community: Pi

Lexers

Pi — Sun, 07 Jun 2026 14:25:27 +0000

what is lexer ?

A lexer is a component that converts raw source code characters into tokens for the parser.

when compiler gets

    .
    .
    total = price + 42;
    .
    .

It doesn't know what it is ? for the compiler it's just stream of characters, without any classification

't' 'o' 't' 'a' 'l' ' ' '=' ' ' 'p' ...

now we have two options, either we make meaning out of this now or during later steps add a extra hurdle to handle each character.

If you are the smart one (unlike me), you will think that handling each character later separately is tedious, so we should handle it now and set some rules about what is allowed and what is not.

Lexer is this small set of instructions which can classify a stream of characters into tokens, not to confuse it with grammar, lexer doesn't know rights and wrongs of language.

if take example of standard english.
jump3d D0g m004. ov3r

The lexer knows that according to the rules we defined, words can't have numbers, so it can't classify this as a valid word.

and if we write
jumped Dog moon. over

The lexer knows this matches the patterns of valid words so it will make Tokens out of this. It still doesn't know if sentence is right grammatically, it doesn't know rules of grammar, it just tries to classify stream of characters for each word.

numbers in identifiers are valid in many languages, the above analogy is just an example to understand the logistics of a language.

These meaningful words in a programming language are called Tokens.

what is a Token ?

Smallest individual element of a program is called as Token. Most things you see inside a program are tokens.

Usually compilers follow the format TokenType(Token) for representation

so total = price + 42 becomes

[IDENT(total), ASSIGN(=), IDENT(price), PLUS(+),NUMBER(42)]

spaces have no value here, there are languages like python which works on indentations and they follow slightly different rules.

So if we take example of

fn add(a,b){
    price = 10;
    total = price + 42;
    print(total);
}

Here the lexer will give output as

[
    FUNCTION(fn),
    IDENT(add),
    LPAREN((),
    IDENT(a),
    COMMA(,),
    IDENT(b),
    RPAREN()),
    LBRACE({),

    IDENT(price),
    ASSIGN(=),
    NUMBER(10),
    SEMICOLON(;),

    IDENT(total),
    ASSIGN(=),
    IDENT(price),
    PLUS(+),
    NUMBER(42),
    SEMICOLON(;),

    IDENT(print),
    LPAREN((),
    IDENT(total),
    RPAREN()),
    SEMICOLON(;),

    RBRACE(})
]

Below is example of Toy lexer

source = """
total = price + 42;
"""

tokens = []

i = 0
while i < len(source):
    ch = source[i]

    # Skip whitespace
    if ch.isspace():
        i += 1
        continue

    # Identifier
    if ch.isalpha() or ch == "_":
        start = i

        while i < len(source) and (source[i].isalnum() or source[i] == "_"):
            i += 1

        value = source[start:i]
        tokens.append(("IDENT", value))
        continue

    # Number
    if ch.isdigit():
        start = i

        while i < len(source) and source[i].isdigit():
            i += 1

        value = source[start:i]
        tokens.append(("NUMBER", value))
        continue

    # Operators
    if ch == "=":
        tokens.append(("ASSIGN", ch))
    elif ch == "+":
        tokens.append(("PLUS", ch))
    elif ch == ";":
        tokens.append(("SEMICOLON", ch))
    else:
        tokens.append(("UNKNOWN", ch))

    i += 1

for token in tokens:
    print(token)

GitHub: https://github.com/pie-314/compiler-blogs/

How to write a compiler ?

Pi — Sun, 07 Jun 2026 14:24:26 +0000

How to write a compiler ?

This series of blogs acts as an analysis and deep dive into compiler design.

Currently I am building EEL. EEL is eBPF Language, this series of blogs consists of all the things I learned throughout the process of building it.

What is a compiler?

A compiler is a program that translates source code written in one language into another representation, usually machine code or an intermediate form that can be executed by a computer.

Modern compilers are made up of several different pieces which come together as a puzzle to make a compiler.

These pieces are:

Lexer
Parser
Semantic Analyzer
IR Generator
Optimizer
Code Generator
Assembler
Linker

We will analyze each of these one by one in great detail.

Source Code
     ↓
   Lexer
     ↓
  Tokens
     ↓
  Parser
     ↓
    AST
     ↓
Semantic Analysis
     ↓
     IR
     ↓
 Optimizer
     ↓
Code Generator
     ↓
 Assembler
     ↓
   Linker
     ↓
 Executable

Throughout the blogs we will take following pseudocode as reference:

fn add(a,b){
    price = 10;
    total = price + 42;
    print(total);
}

GitHub: https://github.com/pie-314/compiler-blogs/

How I Built a Crash-Safe Database Engine in C with Write-Ahead Logging and Snapshots

Pi — Tue, 24 Feb 2026 22:40:39 +0000

Most developers use databases every day. Few actually know what happens when the power goes out mid-write, or when a system crashes halfway through saving data. Yet when the database restarts, everything is still there. That reliability isn’t magic. It comes from careful engineering.

I wanted to understand this at a deeper level, so I built RadishDB. It started as a simple in-memory key–value store in C. Over time, I added persistence, crash recovery, write-ahead logging, snapshots, TTL expiration, a TCP server, and Docker deployment.

The goal wasn’t to compete with Redis. It was to understand how systems like Redis actually work under the hood.

Why C?

Since RadishDB is fundamentally a storage engine, performance and predictability matter a lot. I wanted full control over memory and disk behavior.

C gives you:

direct control over memory
no garbage collector
predictable performance
minimal abstraction between code and hardware

When you're building a database, memory layout and disk writes are not abstract ideas. They are the system itself.

This is also why many real databases like Redis, SQLite, and PostgreSQL are written in C. The language doesn’t hide anything. If something goes wrong, you can usually see exactly why.

It also forces you to think carefully about every allocation, every pointer, and every write to disk.

The Core: In-Memory Storage and Hashtable

RadishDB stores all data in memory. This makes reads and writes extremely fast, since RAM access is much faster than disk access.

To organize data efficiently, I used a hash table.

Hash Tables

Hash tables allow fast lookup, insertion, and deletion, usually in constant time O(1).

When a key is inserted, RadishDB computes a hash and maps it to a bucket.

I used the djb2 hash function:

unsigned long hash(const char *str) {
  unsigned long hash = 5381;
  for (int i = 0; str[i] != '\0'; i++) {
    hash = hash * 33 + str[i];
  }
  return hash;
}

If multiple keys map to the same bucket, they are stored using separate chaining with a linked list.

This keeps operations fast even when collisions occur.

At this stage, RadishDB was fast, but fragile. Everything lived in memory. If the process crashed, all data was gone.

That led to the next problem.

The Problem: Surviving Crashes

An in-memory database is fast, but memory disappears when the process stops.

To solve this, I implemented Write-Ahead Logging (WAL) using an Append-Only File (AOF).

The idea is simple but powerful.

Every write operation is first written to disk before applying it to memory.

For example:

SET name alice
DEL name

These commands are appended to a log file.

If the database crashes, RadishDB reads this file during startup and replays the operations to rebuild memory.

The log becomes the source of truth. Memory becomes a reconstructed state.

This ensures durability.

AOF Rewrite: Log Compaction for Faster Recovery

One problem with append-only logs is that they grow forever.

For example:

SET x 1
SET x 2
SET x 3
SET x 4

Only the final value matters.

Similarly:

SET x 1
DEL x

The key no longer exists, but the log still contains both operations.

Over time, this slows startup and wastes disk space.

To fix this, RadishDB performs AOF rewrite.

Instead of keeping the full history, it writes only the current state into a new file.

The process works like this:

Create a temporary file
Write current database state
Flush to disk using fsync
Atomically replace the old file using rename

Rename is atomic on POSIX systems. This means even if a crash happens during rewrite, the database will always have a valid file.

This ensures both safety and efficiency.

Snapshots: Faster Startup with .rdbx

While AOF is great for durability, replaying a long log can take time.

To solve this, I implemented snapshots using a custom binary format called .rdbx.

A snapshot stores the current state of the database, not the history.

This makes it:

smaller
faster to load
easier to transfer

Snapshots are useful for backups and fast startup.

AOF ensures durability. Snapshots ensure speed and portability.

From Storage Engine to Database Server

At this point, RadishDB could store and recover data. But it wasn’t a real database server yet.

To make it usable by applications, I implemented a TCP server on port 6379.

The server:

creates a socket
listens for client connections
parses incoming commands
executes them
returns responses

The architecture separates responsibilities:

server.c handles networking
repl.c handles command parsing
engine.c handles storage logic

This separation makes the system easier to maintain and extend.

RadishDB became a real database service.

Containerized Deployment with Docker

To make deployment easier, I containerized RadishDB using Docker.

The AOF file is stored in a Docker volume, which ensures data persists even if the container stops.

This makes RadishDB portable and consistent across environments.

It runs the same on local machines, servers, and CI pipelines.

GitHub Actions automate builds and deployment.

Architecture Overview

RadishDB consists of several components:

Engine
Handles in-memory storage, hash table, and command execution.

AOF
Logs every write operation to disk for durability.

AOF Rewrite
Compacts the log by writing only the current state.

Server
Handles TCP connections and client communication.

Docker
Provides consistent deployment and persistent storage.

Each component has a clear responsibility.

What Building RadishDB Taught Me

This project taught me a lot about how databases actually work.

I learned:

how crash recovery works
how write-ahead logging ensures durability
how hash tables work internally
how to design binary file formats
how to build TCP servers
how memory management works in C

More importantly, it changed how I think about systems.

Databases are not mysterious. They are carefully designed systems that follow strict rules to ensure data safety.

Every write, every disk flush, and every recovery step matters.

Conclusion

RadishDB started as a small experiment to understand database internals.

It evolved into a crash-safe database engine with logging, snapshots, networking, and deployment support.

The project helped me understand durability, persistence, and recovery in a practical way.

Building it made databases feel less like black boxes and more like systems built from simple, reliable components.

And that understanding was the real goal.

GitHub: https://github.com/pie-314/radishdb