Hello, I'm Maneshwar. I'm building git-lrc, an AI code reviewer that runs on every commit. It is free, unlimited, and source-available on Github. Star Us to help devs discover the project. Do give it a try and share your feedback for improving the product.
In the previous part, you saw how SQLite takes raw SQL and breaks it into tokens.
At that stage, SQLite has pieces of information, but it still does not understand the meaning or structure of the query.
Now comes the stage where everything starts to make sense.
The parser takes those tokens and organizes them into a structured representation that SQLite can reason about and eventually execute.
What the Parser Actually Does
The parser sits right after the tokenizer in the pipeline. It accepts a stream of tokens and transforms them into a parse tree, which is a structured representation of the SQL statement.
Think of it this way. The tokenizer gives you words, but the parser builds a sentence with grammar and meaning.
For example:
SELECT name FROM users WHERE age > 25;
At the tokenizer level, this is just a sequence of tokens.
At the parser level, this becomes a structured tree that clearly defines what is being selected, from where, and under what condition.
The parser is also responsible for ensuring that the query is valid, both syntactically and semantically.
It checks whether the SQL statement follows the correct grammar and whether the referenced tables and columns actually exist.
The Building Blocks of the Parse Tree
SQLite builds its parse tree using a set of core data structures. These structures are the foundation of how SQL is represented internally.
Token
A Token represents a single unit from the SQL input. It carries the actual text value, such as a literal, table name, or column name. The tokenizer produces these, and the parser consumes them.
The tokenizer output looks like this:
Each piece you see in that output is wrapped as a Token object and passed into the parser for further processing.
Expr
An Expr represents a single operator or operand within an expression. When combined, multiple Expr nodes form a tree that represents a complete expression.
For example, in:
age > 25
age, >, and 25 are all represented as parts of an Expr tree.
ExprList
An ExprList is a collection of expressions. Each expression in the list can optionally have an identifier and sorting information such as ascending or descending order.
You typically see this in SELECT clauses or ORDER BY clauses.
IdList
An IdList is simply a list of identifiers. These identifiers could be column names or other named entities.
For example:
INSERT INTO users (id, name, age)
The (id, name, age) part is stored as an IdList.
SrcList
A SrcList represents data sources. These could be tables, views, or even subqueries. Essentially, anything that can produce rows of data is considered a source.
For INSERT, UPDATE, and DELETE statements, this list usually contains a single source. For SELECT queries, it can contain multiple sources, especially when joins are involved.
Select
The Select structure represents a full SELECT statement. This is especially important for handling subqueries, where one SELECT is nested inside another.
How These Structures Represent SQL
These data structures are combined to form a complete parse tree for different types of SQL statements.
For example, a DELETE statement can be represented as:
DELETE FROM (srclist) WHERE (expr);
An UPDATE statement looks like:
UPDATE (srclist) SET (exprlist) WHERE (expr);
Or more explicitly:
UPDATE (srclist) SET (id=expr, id=expr, ...) WHERE (expr);
An INSERT statement can take two forms:
INSERT INTO (srclist(idlist)) VALUES((exprlist));
or
INSERT INTO (srclist(idlist)) SELECT ...
A SELECT statement combines multiple structures:
SELECT (exprlist) FROM (srclist) WHERE (expr) ORDER BY (exprlist);
Each part of these statements maps directly to one of the core structures like Expr, ExprList, IdList, or SrcList. This mapping is what allows SQLite to understand the intent behind the query.
Syntax and Semantic Validation
The parser does more than just build a tree. It also validates the query.
First, it checks syntax. If the SQL statement is malformed, the parser will reject it immediately.
Then it performs basic semantic checks.
It verifies that referenced tables exist and that the columns mentioned in the query belong to those tables.
This prevents invalid queries from progressing further into execution.
The Parse Object: Carrying Context Through the System
When SQLite begins parsing, it creates a Parse object. This object represents the entire parsing context for a single SQL statement.
This object is passed through all parsing routines and carries global information required during the parsing process.
One of the most important things inside the Parse object is a pointer to a Vdbe object.
Connection to Bytecode Generation
At this stage, the Vdbe object exists but is empty. It is essentially a container waiting to be filled with bytecode instructions.
As parsing progresses, and later when code generation kicks in, this object starts getting populated with instructions that represent the query.
This tight coupling between parsing and code generation is one of the reasons SQLite is so efficient. Instead of building large intermediate representations, it incrementally moves toward executable bytecode.
If you explore SQLite’s source code, especially the build.c file, you can see how parsing and code generation work together, particularly for operations like creating tables and indexes.
In the next part, we will look at how this structured representation is converted into executable instructions, focusing on the Code Generator and name resolution, where SQLite turns understanding into action.
*AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.
git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.*
Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.
⭐ Star it on GitHub:
HexmosTech
/
git-lrc
Free, Unlimited AI Code Reviews That Run on Commit
| 🇩🇰 Dansk | 🇪🇸 Español | 🇮🇷 Farsi | 🇫🇮 Suomi | 🇯🇵 日本語 | 🇳🇴 Norsk | 🇵🇹 Português | 🇷🇺 Русский | 🇦🇱 Shqip | 🇨🇳 中文 |
git-lrc
Free, Unlimited AI Code Reviews That Run on Commit
AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.
git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.
See It In Action
See git-lrc catch serious security issues such as leaked credentials, expensive cloud operations, and sensitive material in log statements
git-lrc-intro-60s.mp4
Why
- 🤖 AI agents silently break things. Code removed. Logic changed. Edge cases gone. You won't notice until production.
- 🔍 Catch it before it ships. AI-powered inline comments show you exactly what changed and what looks wrong.
- 🔁 Build a…
Top comments (0)