Why?
With AI agents so easily accessible online why bother trying to create your own local agent?
There are some good practical reasons, such as having more control over your data, experimenting without worrying about accidentally burning through tokens, and being able to try different models without relying on an online service.
But really: why not.
Don't you need expensive hardware?
A top end GPU and hundreds of Gigs of super fast ram would be nice but for simple experimentation you can get away with a lot less.
This was my setup:
- ZimaBoard 2 with 16Gigs of RAM, and a 160GB SSD
- Proxmox installed onto the ZimaBoard
- A VM with 10GB ram allocated and 40GB disk, running Debian 13
So: no dedicated GPU and limited resources. Running a VirtualBox VM on your desktop would likely give similar results.
I wanted the setup contained. There are Ollama installers for Windows and macOS, but personally I wouldn't recommend experimenting with agents directly on your normal desktop PC, where they might end up having access to more than you expected.
Models are generally described by the number of parameters they have. As a very rough starting point, an unquantized model using 16-bit weights would need around two bytes per parameter. So a 4-billion-parameter model would be roughly:
4 Billion * 2 bytes ≈ 8GB
Searching on Hugging Face gives us plenty of options to experiment with.
Getting to your first Agent Prompt.
First, update the OS and install a few useful utilities:
sudo apt update
sudo apt upgrade -y
sudo apt install curl htop git -y
Then follow the install instructions from Ollama's site.
Note the installer does need root privileges.
curl -fsSL https://ollama.com/install.sh | sh
Once done you can confirm its installed with:
ollama --version
Then you are ready for your first prompt starting with qwen3:4b
ollama run qwen3:4b
Ollama will automatically pull the model the first time it runs, and cache it locally.
You can now ask your first question.
Trying it out.
Starting with a simple one: "how many R's in strawberry"
>>> how many R's in strawberry
Thinking...
First, the question is: "how many R's in strawberry". I need to figure out how many times the letter 'R' appears in
the word "strawberry".
Let me write down the word: S-T-R-A-W-B-E-R-R-Y.
Now, I'll go through each letter one by one and count the 'R's.
- Position 1: S - not R
- Position 2: T - not R
- Position 3: R - yes, R (first R)
- Position 4: A - not R
...
Now, the R's are at positions 3, 8, and 9.
- Position 3: R
- Position 8: R
- Position 9: R
That's three R's.
But let me double-check. Sometimes people might think of it differently, but in English, "strawberry" has three R's.
I can think of the word as a string.
S T R A W B E R R Y
Let me write it out: s-t-r-a-w-b-e-r-r-y
Now, count the 'r's.
- After 't', it's 'r' (first r)
- Then later, after 'e', it's 'r' and another 'r' (so two more r's)
In the sequence: from left to right.
Index: 1:s, 2:t, 3:r, 4:a ....
...
It carried on like that for another 10 minutes or so, thinking, before finally producing the result:
...done thinking.
The word "strawberry" is spelled as follows:
S-T-R-A-W-B-E-R-R-Y.
Counting the letter "R" (case-insensitive, but the word is typically lowercase):
- Position 3: R
- Position 8: R
- Position 9: R
This gives a total of **3 R's**.
The double "R" in "berry" (positions 8 and 9) is standard in English spelling, and "straw" contributes one additional
"R" (position 3).
Thus, there are **3 R's** in "strawberry".
It got there in the end.
Not all of the models expose their thinking like this. It's an interesting insight though, especially when you can see it getting into a loop.
This is just one model, and with Ollama we can simply switch to an alternative:
$ ollama run llama3.2
>>> how many R's in strawberry
There are 2 R's in the word "strawberry".
>>> Send a message (/? for help)
...well, at least it was faster. :-)
10 Gigs of ram is still a lot.
We can go a lot smaller. For example qwen3:0.6b. At 0.6 billion parameters, the model is much smaller and runs considerably faster.
$ ollama run qwen3:0.6b
>>> tell me about your abilities
Thinking...
Okay, the user asked about my abilities, so I need to explain my strengths in a helpful way. Let me start by
acknowledging their interest in me. I should mention my strengths in various areas like tech, communication, and
problem-solving. It's important to highlight how I can support different goals. I should make sure to keep the tone
positive and approachable, avoiding any negative aspects. Let me structure this step by step to cover all the points
without being too lengthy.
...done thinking.
My strengths include a strong focus on **technical expertise** in areas like coding, data analysis, and
problem-solving. I also excel in **communication** and **collaboration**, making me efficient in team settings. I
thrive in **creative environments** and **ambitious goals**, ensuring I can support a wide range of objectives. Let
me know if you have a specific area you'd like me to emphasize!
>>>
It's obviously not going to compete with the huge models running on racks of GPUs, but that's not really the point. Being able to run an LLM of any useful size on a small, low-power machine is interesting in itself.
Summary
Setting up your own LLM for experimentation is quick and easy if you have just a few spare resources to allocate. Using Ollama to manage the models you the flexibility to switch between models and parameter sizes to see for yourself how differently they perform.
Are they practical? Time will tell.
For one final experiment, let's see how the little 0.6-billion-parameter model does at summarising this very article:
...done thinking.
**Summary**
The article outlines how to set up a local LLM (Large Language Model) experiment using Ollama, a tool for running
models locally. The setup involves a Zimaboard with 16GB RAM and a 160GB SSD, along with a Proxmox VM running Debian
13. The user tested a model, "qwen3:4b," by running it on a desktop, which used a VM with 10GB RAM and 40GB storage.
The article emphasizes the practicality of setting up without expensive hardware, noting that 16GB RAM is sufficient
for experimentation. The user also mentions switching to other models (e.g., "qwen3:0.6b") for different parameter
sizes, and highlights the example of testing the "strawberry" word count. The summary concludes that Ollama offers
flexibility for local experimentation with minimal resources.
>>> what would you write as a one or two sentence conclusion?
...
**Conclusion:** Setting up a local LLM with Ollama is practical and efficient, leveraging 16GB RAM and a VM for
experiments like analyzing the "strawberry" word count, which demonstrates the model's capabilities.
Not bad for the little model.
Top comments (0)