DEV Community: Geremi Wanga

Demystifying The Lexer

Geremi Wanga — Wed, 29 Jul 2026 14:32:29 +0000

The Pipeline Behind compiling software can be challenging to grasp... I'll try to break it down as much as i can

First of all the full pipeline looks something like this:

source code → [Lexer] → tokens → [Parser] → AST → [Semantic Analysis] → annotated AST → [LLVM IR] → optimized IR → machine code

The Lexer

The lexer's job is more mechanical. It's also known as the tokenizer.
It reads code character by character dividing it into small items known as tokens which are small chunks of a program, Take this as an example:

let x = 3 + 4;

As raw text, that's just a sequence of characters

l, e, t, a space, x, and so on.

The lexer's job is to turn that into a clean stream of tokens:

After analysis from the lexer:

KEYWORD(let) IDENT(x) EQUALS NUMBER(5) PLUS NUMBER(3) SEMICOLON

Notice that the white space disappeared this is because after analysis they are not needed

Another thing is each chunk or Token got tagged with what kind of a token or type it is

One more thing i found interesting about the lexer is that is uses Regular Expressions (regex)

But how?
you might ask

Simple, The lexer uses it to convert raw character sequences into tokens which are then passed to a passer to verify against the BNF/EBNF which are formal notations systems used to define the syntax of grammar of programming languages

RAW TEXT INPUT: "if ( x > 10 )"
              │
              ▼
 ┌─────────────────────────┐
 │         LEXER           │ ◄── Powered by REGEX (Matches individual character patterns)
 └─────────────────────────┘
              │
              ▼ TOKENS: [KEYWORD("if"), LPAREN, IDENTIFIER("x"), GT, NUMBER(10), RPAREN]

plaintext

The Parser

The parser also knows as a tree builder it builds a tree structure that captures how they relate to each other the (AST), the abstract syntax tree

How does the parser build a tree lets see

A token stream on it's own has no structure

[KEYWORD("if"), LPAREN, IDENTIFIER("x"), GT, NUMBER(10), RPAREN]

The parser builds the tree by consuming the tokens outputted by the lexer

But it does this step by step,

Lexical analysis, Code text is split into small
pieces called tokens
Grammar, it reads these tokens to make sure they follow the strict rules of the language used
Node creation, The parser builds memory objects for operations, variables and statements linking them together into a TREE shape

If we had this as an example
3 + 2 * 5

In the end we will have this as the tree

plaintext

Semantic Analysis

Maybe you've heard of it before in Natural Language Processing (NLP) it refers to resolving meaning and ambiguity in natural language like figuring out if rock means a stone or music genre.

In compilers, it means something narrower and less fuzzy

The AST from the parser only tells you the code is grammatically valid it doesn't know whether x was declared before it's used or whether you're trying to add a string to an integer.

Sematic analysis does this:

Type checking is x + y valid if x is an int and y is a string it might Parse but its still a semantic error
Scope resolution was x actually declared before this line uses it? if there are nested scopes which x does this one refer to?
Declaration calling a function with the wrong number of arguments referencing undeclared variable

LLVM and LLVM IR

LLVM was probably the most complex bit of this expedition one thing i noticed is

LLVM doesn't know or care about your language.

It's a reusable, language agnostic backend for code generation and optimizing Frontends for C, C++, Rust, Swift and many more funnel into it your job as a language author is to translate your AST into LLVM IR

Front-end (Clang, Rust, your language) → LLVM IR → Back-end (x86, ARM, RISC-V, WASM)

plaintext

Back then before LLVM adding a new language or creating your own language with good performance meant writing your own optimizer and codegen for every architecture as unbelievable as that sounds its true.

LLVM turns that N * M problem (language * architecture) into N + M write the optimizations once and they benefit every language

I also wondered what the IR looked like something like this

; Declare the global string constant containing "Hello, World!\n"
@.str = internal constant [15 x i8] c"Hello, World!\0A\00"

; Declare the external printf function signature
declare i32 @printf(ptr, ...)

; Define the main function entry point
define i32 @main() {
entry:
    ; Call printf by passing a pointer to the string
    %call = call i32 (ptr, ...) @printf(ptr @.str)

    ; Return 0 to indicate successful execution
    ret i32 0
}

Polymorphism

Geremi Wanga — Sat, 23 May 2026 08:03:00 +0000

In the last post we talked about encapsulation today we are getting into polymorphism.

An analogy

so we have our houses right. We have a stone house, a brick house, a beach house. They are all houses. But if you asked each one to describe itself, each one would describe itself differently.

The beach house mentions the deck, the brick house mentions the bricks, the stone house mentions the stone.

Same question, different answers. That's polymorphism.
The word itself just means many forms. In Java the same method can behave differently depending on which object is calling it

We can have a describe method on the parent house class and every child class can override it to describe itself in its own way.

You call the same method, you get different results based on what you're actually dealing with.

But why

imagine you have a whole neighbourhood of different houses and you want each one to describe itself. Without polymorphism you'd have to know exactly what type of house each one is and handle each separately.

With polymorphism you just say describe yourself and each house handles it on its own terms. Your code stays clean, you don't have to think about every possible case.
It all connects

This is where everything we've covered starts clicking together. You have a parent class, child classes inheriting from it, fields locked down through encapsulation, and now methods that behave differently depending on who's calling them.

These aren't separate ideas, they're all working together.
The more i go through Java the more i realize these concepts aren't things you just learn and move on from.

They keep showing up together, building on each other, which honestly makes them easier to remember the further in you get.

At this point Java is just becoming my second language

You can't see me

Geremi Wanga — Fri, 22 May 2026 08:45:55 +0000

In the last we talked about inheritance today we are getting into encapsulation

Encapsulation

So we have our house right. Now imagine you have a boiler room in that house the boiler controls the heating and hot water for the whole house.

You don't want just anyone walking in there and messing with it. So you lock the door.

But you still need a way to control the heating so you put a thermostat on the wall, anyone can use the thermostat but nobody can get directly to the boiler.

Thats encapsulation.
In Java you can hide the fields of a class from the outside, make them private so nothing outside that class can directly touch them

But you still provide a way to interact with them through methods. Those methods are called getters and setters. Getters let you read the value, setters let you change it.

But Why

Because direct access is dangerous. If anything in your program can just reach into a class and change its fields directly, things break in ways that are hard to track down.

Encapsulation puts you in control. You decide what can be seen, what can be changed and how.

You are the one setting the rules
That's really what encapsulation is about. You built the house, you decide which rooms are locked and which ones are open.

The thermostat is there for a reason it's a controlled way in. The more i learn Java the more i see that a lot of these concepts are really just about control and structure.

Encapsulation is probably the clearest example of that.
These are just more things i'm picking up as i go. Analogies make it stick better than any definition ever could.

Hand me down an object

Geremi Wanga — Thu, 21 May 2026 21:03:55 +0000

In my last post we were looking at classes..
but today we will get into an concept of an object Inheritance

Objects

An object is an instance of a class. Remember we had a blueprint and an actual house using that blueprint we build a house (object) funny thing is you can build as many houses as you want with the same blueprint this is how object work

Inheritance

Now what if you wanted to build a house but of different kinds for example stone built house or a brick house
a brick house can have the same thing that the stone house has for example the floors, tiles and even color but it has extra things on top like Bricks instead of Stone

But if you already have a blue print for a stone house we don't need to create another separate print for creating a brick house because there's so much information (fields) and even functions (methods) that both the brick and stone house have of course here is where Inheritance Comes in.

In Java a class can inherit everything from another class here there is a child class and a parent class.

Without inheritance we'll be rewriting the same blueprints all the time.

With inheritance we can stand on what already exists modify it to work a little differently like add a few rooms in the brick house from the stone house. Even overwrite a method by changing what it does instead of rebuilding from the ground up again.

The more I learn Java the more i did deeper into Object oriented programming and how things can be connected

Inheritance is a means of connection of classes. Classes either build on top of one another or they relate to each other. The child knows what the parent knows like a hand me down of methods an fields.

This is what I think in Java in general everything is connected to something else there is no such thing as separate concepts something can be connected to something else

Settle in Class

Geremi Wanga — Thu, 21 May 2026 20:14:58 +0000

In regular Object oriented programming for example in JavaScript It's is Prototype based way meaning it uses objects that inherit directly from other objects.

Java does things differently. Unlike other object oriented languages Java is class based not prototype based meaning everything you write in Java lives inside of a class.

What does that mean?

Class

A class is like a blueprint you use before you build a house, You will build a house using this blueprint later on
and the house is what we define as an object

What does a class know

Here is where we introduce something inside the class called a field. A field going back to our previous analogy is like describing the house or object for example color of the house number of rooms things of the sort

Constructors

When you need people to start working on the house before you get in you can tell them or not once you give them the plan (class) and information about the house (fields) they'll get to work even without telling them anything
The same thing happens with constructors when you create an object a constructor runs automatically.

The constructor is to get everything ready even before the start. Instead of going piece by piece it does everything once at the beginning

Methods

The methods inside a class are the ones that give your class behavior. They act like functions defined inside of a scope and you can only access those function by referencing the scope they are in. in our analogy a method is like a field that performs tasks with the data for example a method that calculates the area of the house sits in

These are just some of the few concepts i learned in Java i try using analogies and associate them to real life so i can remember them later on..
class dismissed

Windows Wearing Hoodies

Geremi Wanga — Thu, 21 May 2026 14:07:43 +0000

Back then before 2016 people always saw linux users..
They wore hoodies and they looked happy because of how free they were. They got jealous and so Microsoft invested a significant amount of resources in creating a virtual environment inside windows that runs linux commands and is executed by the windows operating system this was WSL 1

First they started with the idea of having a windows command translated directly into

Fast forward, and now we have WSL 1 and WSL 2

But have you Ever wondered what happens when you run Linux commands via WSL? Let’s break it down.

In one of last posts i explained the concept of a system call and the journey from shell all the way to the hardware

A system call is how a program asks the operating system to do something for it.

The first one uses a system call translation layer converts Linux system calls into Windows kernel calls.
This interestingly works without any need for a Virtual machine.
The only problem with WSL1 was poor file system performance
But still It was proved to be possible

Then the Second Came for this one it actually runs a real Linux kernel inside a lightweight Hyper V Virtual Machine.
Linux commands execute here without any traslation layer with full system call support. Crazy right

fun thing about wsl2 is it Supports Docker containers

JavaC

Geremi Wanga — Thu, 21 May 2026 13:35:05 +0000

Normally languages like C and C++ compile their code directly into machine code to be executed by the machine.

This machine code is specific to the machine and operating system you are executing in and sometimes it may not work.

Now here's why JavaC is better than GCC or any C compiler.

Java has an intermediate step during compiling it.
It converts Java file into another human readable .java file into ByteCode which is a format that is neutral and readable across many operating systems unlike GCC.

JVM

This is Java's virtual machine.. In my last post we saw how an command is taken from shell all the way to the kernel to the Hardware using a system call now instead of this we execute the code using this virtual machine called JVM now this sits on top the operating system

But how? There is a specific jvm version for all operating systems for almost all operating systems Now the jvm converts this byte code into machine code that the specifc machine understands or system calls that the machine can use

Evolving

The concept of build once run anywhere has now evolved tools like docker and other containerization platforms have adopted this principle and have used it to build working applications that run anywhere

In the next post we are going to dig deeper into virtual machine and the most popular virtual machine WSL

Past the JVM

Geremi Wanga — Thu, 21 May 2026 12:48:15 +0000

Learning Java got me thinking how exactly does things word under the hood.. I'm not talking about Java in Particular I'm mentioning programming in General how does a command I Write on my shell or bash terminal get executed in the machine

What role does the kernel play in all of this
And who does all the heavy lifting internally is it the kernel or the hardware

I did research on all of this let me walk you through it

The user and the shell

When I type a command, the shell interprets it and decides which actions to take.

System Calls

The shell issues system calls eg fork(), execve(), and open() to request services from the kernel.

Kernel to Hardware

The kernel handles the system calls, communicates with device drivers, and interacts with hardware safely and in a controlled way.

Back to user

The output appears in the terminal all of this happens in milliseconds

What so great about all this is how every user program, not just the shell, uses system calls to communicate with the kernel. It’s like a controlled gateway to the operating system

I'd like to do more research about this Java in particular this time we'll go under the

Galileo Gosling the father of Java

Geremi Wanga — Wed, 20 May 2026 09:39:51 +0000

Before Getting into development straight away.
We must learn the origin the customs and the way of Java Right from the start.

We must start first with the man behind it.

The Father of Java James Gosling was born in 1955 in Canada and later Joined Sun Microsystems in 1984 where he worked for 26 years.

In 1994 He got the Idea while writing a program to port software from a PERQ by translating Perq Q-Code to VAX assembler you know just normal stuff

He created the original design of Java and implemented the compiler (JAVAC) and also the Virtual machine (JVM)

Though it wasn't all sunshine. Java was acquired later on by Oracle from sun.

After this James Gosling decided to part ways with sun microsystems in April 2 2010. Since then he has taken a critical stance toward Oracle.

In march 2011 he joined Google for six months.
After he followed his friend Bill Vass and joined into a startup Liquid Robotics. Things were doing better for Gosling until the startup became too good, later in 2016 Liquid Robotics was acquired by Boeing.

Following the inevitable Mr Gosling left Liquid Robotics and joined AWS in 2017 as a Distinguished Engineer and later Retired in 2024

Being A creator comes with a lot of hardships and mishaps. Even though Gosling had differences with oracle because of being stripped off decision making authority. People still associate him Greatly with the JAVA language He is still the father of Java

Aside from all this he was still accredited for a lot of things
In

2002: awarded The Economist Innovation Award
2002: awarded The Flame Award USENIX Lifetime Achievement Award.
2007: made an Officer of the Order of Canada.[32] The Order is Canada's second highest civilian honor. Officers are the second highest grade within the Order.
2013: became a fellow of the Association for Computing Machinery.
2015: awarded IEEE John von Neumann Medal
2019: named a Computer History Museum Fellow for the conception, design, and implementation of the Java programming language

Even with all the challenges Gosling remains an inspiration to many for his resilience including myself

Head First Java

Geremi Wanga — Tue, 19 May 2026 12:27:41 +0000

One of the best things when learning a new language would be to read.

Read and understand the vocabulary...

Occasionally write with that language this helps build muscle memory and memory.

But for JAVA this is entirely different you can't read syntax then use it later on you have to grasp the syntax the Classes, you have to jump straight into the "void" 😆 so to speak

But as the famous Ray BradBury said 'Jump, and you will find out how to unfold your wings as you fall.'

But what are the wings in Java? That would be the resources and for me the best resource I've found so far is Head First Java

written by Kathy Sierra and Bert Bates
With photos and visuals it has been engineered to keep your attention Pinned and for People with ADHD 🤫 you won't find any trouble paying attention

If you're also learning Java definitely check out this book

The coffee language

Geremi Wanga — Mon, 18 May 2026 10:38:58 +0000

Sometimes learning a new Language can be hard...Believe me it's not as hard as Java

There's a reason why most beginners quit after the first few tutorials of learning 'The learning curve' is extreme
Most senior engineers say that the initial bit of Java is the hardet to climb but after every hill is a landslide

The Syntax

This is the number one reason why beginners quit learning Java with it's strong Object Oriented syntax... It's a hard mountain to climb
Concept like:

Encapsulation
Inheritance
Polymorphism Being the hardest bones to chew

For the next week i'll be sharing my progress and learning materials

If you are also starting your journey on learning Java.. Join me as we strive to conquer one of the hardest mountains created by man James Gosling 😉

Stay tuned

Building Klimatt

Geremi Wanga — Mon, 18 May 2026 08:11:29 +0000

A smarter way to track farm markets in Kenya

Over the past few months I’ve been building Klimatt, a farming platform focused on helping farmers make better decisions using market data and simple management tools.

A big part of the project started after I found food price datasets from Kenya provided through WFP/FPMA records. I began experimenting with parsing the CSV data using Go and exposing it through APIs to a React frontend.

The idea slowly grew into something bigger.

Current features

📊 Market price tracking from markets across Kenya
🌽 Commodity and stock keeping
🛒 Marketplace where farmers can post and discover products
📅 Farming calendar for tracking activities like planting schedules and rainfall periods

Working with real agricultural data

One of the most interesting parts has been working with real agricultural datasets:

normalizing prices
grouping commodities by region
comparing markets
calculating trends

Example:

router.GET("/api/prices/latest", func(c *gin.Context) {
    c.JSON(200, response)
})

Tech stack

Go + Gin
React + TypeScript
CSV ingestion pipelines
n8n automation experiments

I recently made a short Loom walkthrough showing the direction of the project:

Still a work in progress, but it’s been exciting building something around a real problem affecting farmers and market access.

Would love feedback from anyone working in:

agri-tech
logistics
data engineering
market systems
Go backend development