Why I started this project
I wanted to understand how AI agents actually work, beyond just using a framework.
So I decided to build a small one from scratch in Python.
It started as a personal learning exercise. I wasn't trying to build something production-ready or create another agent framework. I just wanted to understand the different components, how they interact, and what happens when you add more capabilities.
I eventually ended up with nine versions of the same project, each introducing one new architectural concept.
After finishing it, I thought it might be useful to share the process with anyone interested in learning how agents work.
Starting with simple rules
The first version was intentionally basic.
It could receive a request, choose a tool using predefined rules, extract arguments, execute the tool, and store the result.
Nothing fancy. No LLM, no external API, no framework.
But that simplicity made it easier to see what each component was actually doing.
From there, I started separating responsibilities. First the Executor, then a more generic Parser, followed by planning, memory management, and a Tool Registry.
My approach was simple: one version, one concept, one architectural improvement.
Adding an LLM
Only after building those components did I introduce a language model.
I used Qwen2.5-0.5B-Instruct, running locally, first for choosing tools and later for extracting arguments from natural language.
What I found useful was that I could replace the rule-based components without having to redesign the entire system.
The rest of the architecture stayed mostly the same.
It helped me understand that the LLM is just one part of the agent. The surrounding code still handles tool execution, validation, planning, and state.
From one step to multiple steps
The final version introduced basic multi-step execution.
For example, given this request:
Add 2 and 3, then multiply the result by 4.
The agent creates two steps, executes the addition, and passes the result to the multiplication step.
The final answer is 20.
It's a small example, but it brings together the components developed throughout the project.
What I learned
The biggest takeaway for me was that building an agent became much less confusing once I stopped treating it as one big system.
Separating routing, parsing, execution, planning, and memory made it easier to understand what each part was responsible for.
It also made frameworks like LangChain or CrewAI easier to approach, because I now had a clearer idea of the problems those frameworks are trying to solve.
There are still plenty of limitations. The planner uses simple rules, memory is stored in a Python list, and the local LLM can make mistakes when choosing tools or extracting arguments.
But that's fine for what I wanted to learn.
Sharing the project
I've made the whole project available on GitHub, with all nine versions, Jupyter notebooks, architecture diagrams, and documentation in English and Italian.
There's also an executable notebook on Kaggle.
- GitHub: https://github.com/lucalullo/building-ai-agent
- Kaggle: https://www.kaggle.com/code/lucalullo/building-ai-agent
I originally built this for myself, but I hope the step-by-step approach might help someone else understand agent architecture a little better.
If you've tried building an agent without a framework, I'd be interested to hear how you approached it.
Writing note: I used AI assistance to translate, organize, and polish this article in English. The project, its implementation, and the technical content are my own.
Top comments (0)