What I Learned After Building a Programming Language From Scratch in Python

I didn't just write a programming language. I built the pipeline that makes the language work.

🔗 NexPro on GitHub: https://github.com/probal2005/NexPro

There are thousands of programming languages in existence.

So building another one sounds unnecessary.

But my goal with NexPro was never to compete with Python, JavaScript, Rust, or C++.

I wanted to answer a much simpler question:

What actually happens between writing code and getting an output?

So instead of only learning the theory of lexers, parsers, ASTs, interpreters, and runtimes, I decided to implement them.

This post documents what I have actually built, what works today, what I learned, and what still needs to be done.

---

1. The Experiment

NexPro is an experimental programming language implemented in Python.

Its source files use the:

.pa

extension.

A simple NexPro program looks like this:

name = "Probal"
city = "Kolkata"

say name
say city

The important part isn't that this syntax is simple.

The important part is what happens internally.

The source doesn't go directly from:

.pa file

to:

output

Instead, it passes through multiple stages.

---

2. The Actual Language Pipeline

The core idea behind NexPro is:

┌───────────────────┐
│   NexPro Source   │
│      .pa          │
└─────────┬─────────┘
          │
          ▼
┌───────────────────┐
│       Lexer       │
│  Source → Tokens  │
└─────────┬─────────┘
          │
          ▼
┌───────────────────┐
│      Parser       │
│ Tokens → AST      │
└─────────┬─────────┘
          │
          ▼
┌───────────────────┐
│        AST        │
│ Program Structure │
└─────────┬─────────┘
          │
          ▼
┌───────────────────┐
│    Interpreter    │
│   Execute Nodes   │
└─────────┬─────────┘
          │
          ▼
┌───────────────────┐
│      Runtime      │
│ Values & State    │
└─────────┬─────────┘
          │
          ▼
┌───────────────────┐
│      Output       │
└───────────────────┘

This pipeline isn't just a diagram for documentation.

These are the actual conceptual components implemented in the repository.

---

3. Evidence #1 — The Repository Has a Language Implementation Structure

The project is organized around the language itself:

NexPro/
│
├── nexpro/
│   ├── cli.py
│   ├── lexer.py
│   ├── parser.py
│   ├── interpreter.py
│   ├── runtime.py
│   ├── tokens.py
│   ├── ast.py
│   ├── errors.py
│   └── __version__.py
│
├── examples/
│   ├── hello.pa
│   └── variables.pa
│
├── tests/
│
├── README.md
├── LICENSE
└── pyproject.toml

Each part exists for a reason.

Component	Responsibility
lexer.py	Converts source text into tokens
tokens.py	Defines token types
parser.py	Builds program structure
ast.py	Represents syntax as nodes
interpreter.py	Executes the AST
runtime.py	Handles runtime behavior/state
errors.py	Language-level error handling
cli.py	Provides the command-line interface
tests/	Tests language behavior

This separation is important because language implementations become difficult to maintain when lexing, parsing, execution, and runtime logic are mixed together.

---

4. Evidence #2 — NexPro Actually Executes .pa Programs

A language project shouldn't stop at syntax diagrams.

It needs to execute programs.

NexPro provides a CLI command:

nexpro run examples/hello.pa

For a program such as:

say "Hello NexPro!"

the interpreter produces:

Hello NexPro!

Variables can also be used:

name = "Probal"
city = "Kolkata"

say name
say city

which produces:

Probal
Kolkata

That gives us a complete path:

hello.pa
   ↓
CLI
   ↓
Lexer
   ↓
Parser
   ↓
AST
   ↓
Interpreter
   ↓
Output

---

5. Evidence #3 — Tokens Exist Before Execution

Consider:

name = 10

A programming language implementation doesn't need to treat this as one giant string.

The lexer can break it into meaningful pieces:

IDENTIFIER(name)
ASSIGN(=)
NUMBER(10)

That transformation is fundamental.

The lexer answers:

"What are the pieces?"

The parser answers:

"How are those pieces related?"

The interpreter answers:

"What should those relationships do?"

This separation is one of the biggest things I understood while building NexPro.

---

6. Evidence #4 — The AST Changes Everything

Consider:

a = 10 + 20

The parser doesn't simply need to remember:

"10 + 20"

It can represent the expression structurally:

        Assignment
        /        \
       a        Binary(+)
                /      \
              10        20

This is the Abstract Syntax Tree.

The AST gives the interpreter a structured representation of what the programmer wrote.

Instead of asking:

"What characters are in this string?"

the interpreter can ask:

"What kind of node am I executing?"

That distinction is fundamental to language implementation.

---

7. Evidence #5 — Arithmetic Becomes a Tree

For:

a = 10
b = 20

say a + b

the important expression is:

a + b

Conceptually:

       +
      / \
     a   b

The interpreter can then resolve:

a → 10
b → 20

and evaluate:

10 + 20

giving:

30

This is where a simple syntax feature starts demonstrating the full language pipeline.

---

8. What Building the Lexer Taught Me

At first, tokenization looks easy.

You see:

a = 10

and think:

Split the string.

But real language syntax quickly introduces problems.

What happens with:

a = 10 + 20

What about:

name = "Probal"

What about:

say "Hello World"

Now the lexer needs to distinguish:

IDENTIFIER
NUMBER
STRING
ASSIGN
PLUS
SAY

It also needs to deal with things such as:

Whitespace
Unknown characters
Strings
Numbers
Identifiers
Operators
End of file

That was one of my first major lessons:

A programming language begins before parsing.

---

9. What Building the Parser Taught Me

The parser is where syntax becomes structure.

For example:

a = 10 + 20

is not equivalent to:

a + 10 = 20

The parser needs to understand relationships and precedence.

Even simple arithmetic starts raising questions:

10 + 20 * 5

Should it mean:

(10 + 20) * 5

or:

10 + (20 * 5)

Language design quickly turns into a combination of:

Syntax
+
Grammar
+
Precedence
+
AST design

That's something I didn't fully appreciate before implementing it.

---

10. What Building the Interpreter Taught Me

The interpreter is where the language becomes executable.

It has to understand things such as:

Number
String
Variable
Assignment
Binary Expression
Say

For example:

x = 100
say x

requires the runtime to maintain state:

Environment

x → 100

Then:

say x

requires the interpreter to retrieve:

x → 100

and send the value to the output.

This is where concepts like scope, environments, values, evaluation, and runtime state become practical instead of theoretical.

---

11. Why I Chose Python

NexPro is currently implemented in Python.

That was intentional.

For an experimental interpreter, Python gives me:

• Fast iteration
• Easy testing
• Simple data structures
• Classes for AST nodes
• Dictionaries for environments
• Straightforward exception handling

I can focus on:

language design
+
lexing
+
parsing
+
AST
+
interpretation

without initially having to solve low-level implementation problems.

That doesn't mean Python is necessarily the final implementation language.

It means Python is a practical place to start.

---

12. Testing Is Not Optional

A language implementation can break very easily.

For example:

Lexer change
     ↓
Token change
     ↓
Parser breaks
     ↓
AST changes
     ↓
Interpreter breaks

That's why NexPro includes tests.

The project has a dedicated:

tests/

directory for language behavior.

As NexPro grows, I want the test suite to cover:

Lexer
  ↓
Parser
  ↓
AST
  ↓
Interpreter
  ↓
Runtime
  ↓
CLI

The goal is to make every new language feature measurable and reproducible.

---

13. What NexPro Can Do Today

The current implementation is intentionally small.

The project has already moved beyond a source-code mockup into an executable interpreter with components for:

✓ CLI execution
✓ Lexical analysis
✓ Tokens
✓ Parsing
✓ AST representation
✓ Variables
✓ Assignment
✓ Strings
✓ Numbers
✓ Basic expressions
✓ Interpretation
✓ Runtime structure
✓ Error handling structure
✓ Tests

The exact supported syntax will continue changing as the language develops.

That distinction matters.

I'm documenting what exists rather than presenting planned features as completed features.

---

14. What NexPro Cannot Do Yet

This is equally important.

NexPro is not yet:

❌ A production compiler
❌ A Python replacement
❌ A high-performance language
❌ A mature standard library
❌ A complete ecosystem
❌ A stable 1.0 language

There are still major areas to develop:

Functions
Loops
Conditionals
Collections
Modules
Type system
Standard library
REPL
Tooling
Debugger
Package management
Performance

And that's exactly why I consider it an interesting engineering project.

---

15. Roadmap

My roadmap is currently divided into three stages.

Stage 1 — Language Core

✓ Lexer
✓ Tokens
✓ Parser
✓ AST
✓ Interpreter
✓ Variables
✓ Basic expressions
✓ CLI

→ Conditionals
→ Loops
→ Functions
→ Collections

Stage 2 — Developer Experience

→ REPL
→ Better diagnostics
→ Formatter
→ Documentation
→ VS Code syntax highlighting
→ Debugging tools
→ Improved testing

Stage 3 — Advanced Runtime

→ Modules
→ Standard library
→ Package system
→ Bytecode
→ Performance improvements
→ Possible compilation

These are goals, not completed features.

---

16. Why This Project Matters to Me

The biggest result of NexPro isn't the syntax.

It's the understanding.

Before building a language, I knew the words:

Lexer
Parser
AST
Interpreter
Runtime

After implementing them, I started seeing programming languages as pipelines.

When I write:

say 10 + 20

I can now mentally see:

SOURCE
  ↓
TOKENS
  ↓
SYNTAX
  ↓
AST
  ↓
EVALUATION
  ↓
RUNTIME
  ↓
30

That shift in understanding is the real reason I started NexPro.

---

17. What I Want to Investigate Next

The next interesting question isn't just:

"What syntax should NexPro have?"

It's:

"How far can I take a language that started as a Python interpreter?"

Some questions I want to explore:

Can NexPro have a real type system?

For example:

age: number = 20
name: string = "Probal"

Can it compile to bytecode?

Instead of:

Source → AST → Interpreter

potentially:

Source
  ↓
AST
  ↓
Bytecode
  ↓
Virtual Machine

Can the language have its own package system?

Something like:

import math

Can it eventually have an IDE experience?

For example:

NexPro
 ├── Language Server
 ├── Formatter
 ├── Debugger
 └── VS Code Extension

Those are future experiments.

---

18. The Repository Is the Evidence

Rather than asking readers to trust a description, I want the repository to be the evidence.

You can inspect:

lexer.py
parser.py
ast.py
interpreter.py
runtime.py
tokens.py
errors.py
cli.py
tests/
examples/

You can run the examples.

You can inspect the implementation.

You can open issues.

You can suggest changes.

You can fork it.

You can try to break it.

That's how I want NexPro to evolve.

---

19. Open Source Means Feedback

I'm especially interested in feedback from developers who have experience with:

Compilers
Interpreters
Programming Languages
Parser Design
ASTs
Python
Developer Tooling
Language Design

Some questions I'd genuinely like feedback on:

Is the current architecture reasonable for an interpreter?

Which language feature should be implemented next?

Should NexPro remain dynamically typed or eventually introduce static typing?

Should the next major milestone be a REPL, functions, or a bytecode VM?

What architectural mistakes should I fix before the project becomes larger?

---

20. Final Result

NexPro started with a simple experiment:

Can I build a programming language?

The answer isn't:

"I built the next Python."

That's not what this project is.

The more accurate answer is:

I built a small, executable programming language implementation, and I'm using it to explore how languages are designed and executed.

And that's already taught me more about programming-language internals than simply reading about them.

The journey currently looks like:

          NEXPRO

       Source Code
            │
            ▼
          Lexer
            │
            ▼
          Tokens
            │
            ▼
          Parser
            │
            ▼
           AST
            │
            ▼
       Interpreter
            │
            ▼
         Runtime
            │
            ▼
          Output

The interesting part is that this is only the beginning.

---

🚀 Try NexPro

If you want to inspect the implementation, experiment with the syntax, or contribute:

🔗 GitHub: https://github.com/probal2005/NexPro

If you find something wrong, open an issue.

If you have an idea, tell me.

If you want to experiment, fork it.

And if you know programming-language implementation better than I do, please tell me what I'm doing wrong.

That's exactly the kind of feedback I want.

NexPro is small today.

But every programming language starts somewhere.

And this is mine.

---

What would you build next?

Functions, a REPL, a type system, or a bytecode VM?

programming #python #programminglanguages #compiler #interpreter #opensource #softwareengineering #parsing #ast #developer