DEV Community: Erick Mwangi Muguchia

No GPU? No problem!, running local AI efficiently on my CPU.

Erick Mwangi Muguchia — Tue, 14 Apr 2026 20:21:12 +0000

1. Why I tried this.

2. My setup.

3. The problems I faced.

4. The tweaks I discovered.

5. Results.

6. Lessons.

THE WHY:

I’ve always wanted to explore deep conversations with AI and understand how these systems work. For a long time, that dream was limited by the lack of a GPU and the high cost of apps that allow meaningful interaction with AI models. Now, I’m determined to overcome those barriers and build my own path into this world.

So the idea of running one locally always kept bugging me, and I finally got to do it.
It was a long and educational journey, so let me walk you through it.

My setup

OS: Arch Linux x86_64
Workflow: tmux + i3 (just because I like using key-bindings), starship + wezterm
Hardware :CPU: Intel(R) Core(TM) i5-7200U (4) @ 3.10 GHz
GPU: Intel HD Graphics 620 @ 1.00 GHz [Integrated]
Memory: 2.75 GiB / 7.61 GiB (36%)
Swap: 666.95 MiB / 3.81 GiB (17%)

Storage: Disk (/): 28.64 GiB / 31.20 GiB (92%) - ext4
Disk (/home): 39.15 GiB / 84.33 GiB (46%) - ext4
Disk (/run/media/shinigami/Vault): 349.96 GiB / 457.38 GiB (77%) - ext4

Ollama is the engine I used to run models locally. By default, it stores models in ~/.ollama, which quickly filled my root partition.
install with:
curl -fsSL https://ollama.com/install.sh | sh

To fix this I redirected the models to a secondary storage.

mv ~/.ollama /run/media/shinigami/Vault/ollama
ln -s /run/media/shinigami/Vault/ollama ~/.ollama

This symlink forces Ollama to store everything on the Hard Drive , solving the disk full error.

Now pulling and managing models.

I started by pulling the smallest model, so I went for the Tinyllama, which is about 637 MB.
After testing the model, > fast yes but not that smart, so I had to look for a bit smarter models.
ollama pull tinyllama

Therefore I went ahead and pulled heavy models like llama3.2:3b, and llama3.2:1b, which were 2.0 GB and 1.3 GB respectively.
ollama pull llama3.2:3b
ollama pull llama3.2:1b

Building custom builds for them
So I wanted maximum output/use of the models so I created custom models using modelfiles.
Example;
I wanted one to help me with learning basic networking concepts, the model files looks like this:


FROM llama3.2:3b # Adjust accordingly

# ---------- PARAMETERS ----------
# Lower temperature for accuracy; moderate top_p for variety without drifting.
PARAMETER temperature 0.25
PARAMETER top_p 0.9

# More room for code + explanations.
PARAMETER num_ctx 4096

# Safer repetition control (prevents looping).
PARAMETER repeat_penalty 1.12

# If your setup supports it and you want more deterministic answers, you can also try:
# PARAMETER seed 42

# ---------- SYSTEM BEHAVIOR ----------
SYSTEM """
You are My Mentor: a concise, practical networking fundamentals tutor and coding assistant.

Primary goal:
- Teach networking fundamentals clearly and correctly (OSI/TCP-IP, IP addressing & subnetting, ARP, DNS, DHCP, TCP vs UDP, ports/sockets, routing, NAT, HTTP/TLS basics, troubleshooting with ping/traceroute/nslookup/curl/tcpdump, basic firewalls).

Secondary goal:
- Produce meaningful, runnable code examples when useful prefer (enter preferred language).

Style rules:
- Explain concepts with short definitions + one concrete example.
- When writing code, include: what it does, how to run it, expected output, and common pitfalls.
- If the user’s question is ambiguous, ask up to 2 clarifying questions before answering.

Output format (default):
1) Concept (2–5 sentences)
2) Why it matters (1–2 bullets)
3) Example (diagram, packet flow, or command)
4) Code (only if it adds value; keep it minimal)
5) Quick check (2–3 questions to self-test)
"""

That snippet explains much of it.

Building the models

After creating the Model file, vim Modelfile1, now we bind it to the any models we downloaded.
A Modelfile is the blueprint that shapes an AI model’s personality, rules, and behavior on top of its base intelligence.

Command for creating from the Modelfile,
ollama create My-model -f Modelfile1

you should see this output:

gathering model components 
using existing layer sha256:dde5aa3fc5ffc17176b5e8bdc82f587b24b2678c6c66101bf7da77af9f7ccdff 
using existing layer sha256:966de95ca8a62200913e3f8bfbf84c8494536f1b94b49166851e76644e966396 
using existing layer sha256:fcc5a6bec9daf9b561a68827b67ab6088e1dba9d1fa2a50d7bbcc8384e0a265d 
using existing layer sha256:a70ff7e570d97baaf4e62ac6e6ad9975e04caa6d900d3742d37698494479e0cd 
creating new layer sha256:7f89bd8bf6ef609a9aefeab288cde09db6c1ef97f649691f25b29e0f85a8c91c 
creating new layer sha256:446b3a23f7599dc79a11cfb03c670091c9fe265aba28fa3316e9e46dc86365db 
writing manifest 
success

"My-model" > you can name it anything.
Plus you can create as many Modelfile as you like giving them different task, and of course you can add more rules and examples in the Modelfile as you like.

After successfully creating 'My-model', run the model,
ollama run My-model

➜ ollama run My-model
>>> Send a message (/? for help)

That was the fun part, now the real problem was now when the models runs and now CPU starts screaming because of 100% CPU consumption, overheating, which ends up making the model work slow.

So I had to optimize my setup to better handle the models.
By using the tools to monitor CPU (htop), and heat (lm_sensors), I was able to better optimize my setup.
To run the models efficiently I had to:

Maximize CPU performance when running the models.
Reduce latency of bottlenecks.
Stabilize thermal behavior.
Prioritize computer-heavy processes.

Running local AI models on CPU introduces:

Lower parallelism.
Thermal Throttling.
OS scheduling inefficiencies.
Power-saving defaults limiting performance.

So instead of forcing the model, You optimize around it.
Unlock CPU performance
In Arch
sudo pacman -s cpupower
then install the tuned for changing CPU frequency state and for system wide optimization.
sudo pacman -S tuned
Then enable it :
sudo systemctl enable --now tuned

Then change the CPU performance;
sudo cpupower frequency-set -g performance

Switches CPU governor: By default, Linux CPUs often run in “ondemand” or “powersave” mode, scaling frequency up and down depending on load.
Performance mode: Locks the CPU to its maximum frequency, ensuring consistent speed.

Impact:
Faster response times for heavy workloads (like tokenization, AI inference, or compiling).
Reduced latency spikes since the CPU doesn’t waste time ramping up.
More predictable benchmarking results.

cons:
Higher power draw, more heat, fans spin up, battery drains faster on laptops.

Then confirm the configuration by,
cpupower frequency-info

➜ cpupower frequency-info
driver: acpi-cpufreq
hardware limits: 400 MHz - 2.60 GHz
available cpufreq governors: conservative ondemand userspace powersave performance schedutil
current policy: governor "performance" within 400 MHz - 2.50 GHz
current CPU frequency: 3.10 GHz (kernel reported)
boost state: Supported, Active

Then apply throughput performance profile, using the 'tuned' we installed earlier.
sudo tuned-adm profile throughput-profile

Why this matters:

Optimizes CPU behaviors.
Improves disk I/O.
Adjusts system scheduling.
Reduces unnecessary power-saving interruptions.

Results: Smoother, sustained compute performance.

Model am using (and why)

llama3.2:3b

Balanced size and capability.
Noticeably smart.
Good for deeper prompts and reasoning.
This felt like middle ground between speed and intelligence.
phi3:mini

Very efficient for its size.
Strong reasoning compared to other small models.
Optimized for lower-resources environments
This one stood out as surprisingly powerful for CPU use.

This concludes my first phase: setting up, tuning performance, and confirming that local AI models run smoothly. In the next phase, I’ll dive into measuring tokenization speed; using verbose logs and custom C scripts to compare how these models perform under different workloads.
"Turns out you don't need powerful hardware to explore AI, just curiosity and a stubborn CPU."

Running local AI.

Erick Mwangi Muguchia — Sun, 12 Apr 2026 15:14:03 +0000

I had this idea to run AI locally on my own laptop. Just to see if I could. Ended up going with Ollama.

At first it was brutal — all CPU, no GPU, super slow. But I messed around, tweaked some stuff, and finally got it to actually run okay. Not fast, but okay.

Then I went down a rabbit hole. I wanted to know what the models were doing. Like, how hot is my CPU getting? How fast is it spitting out tokens? So I started building my own little monitoring setup. Used C for some low-level stuff, Dash for a live dashboard, Python to glue it all together. Oh and lm-sensors to watch the temps because this thing makes my laptop sweat.

Now I can sit there and watch my models run in real time. Token rate, memory, core temps — all on a dashboard.

Feels good having AI running offline. No cloud, no weird latency, just my machine. And a bunch of scripts I broke and fixed along the way.

If you're thinking about trying local AI, just go for it. Just know you'll end up tinkering way more than you expect. Worth it though.

Web apps. I like making web apps ..for celebrations or for fun. So its christmas and I made a small web app. And I didn't use html or js or css... But i used C programming to make it.. It was stressing but its good and am happy about it .

Erick Mwangi Muguchia — Tue, 23 Dec 2025 04:30:38 +0000

I made a promise to myself that am not leaving Meru University without Python skills.

Erick Mwangi Muguchia — Fri, 12 Dec 2025 09:36:29 +0000

When I arrived at Meru University, I made myself a deal:
"I will not leave this place without learning Python."

The first thing I did was relocate. I needed to minimize distractions and move to an environment conducive to focused learning.

Why Python?

I'd heard so much about it—web development, data science, AI, endless possibilities. I was determined to master it and open doors in tech. But I had zero programming knowledge. I knew it would be challenging, but I was willing to put in the effort.

Month 1: Building Foundations

I downloaded tutorials, read documentation, binged YouTube. I learned Python syntax, data types, control structures. Then I practiced—a lot. Small programs. Number games. These games weren't just practice; they made learning fun. That mattered more than I expected.

Month 2–3: Going Deeper

After a month, I decided to add complexity. I wanted to understand how programming actually works, not just write code. So I added C to my learning path.

This wasn't random. Python was my safety net. C forced me to understand memory, pointers, how computers actually think. It made Python click in a new way.

Learning both simultaneously was hard—but it worked.

Month 4: The Full Picture

As weeks turned into months, I got proficient in both. I signed up for GitHub's student pack (more resources, better tools). I learned version control—essential for any real programmer.

Then came R for statistical programming and data visualization. Each language opened new doors.

The Progress

Now, as the semester ends, I can say this honestly: I've made significant progress.

I have:
✅ Multiple projects built and on GitHub
✅ Mastery of Python, C, and R
✅ Understanding of version control and collaborative development
✅ A journaling habit that tracked every step

The Real Win

Learning these languages wasn't just about syntax. It boosted my confidence. It showed me I can learn anything if I commit to it.

And I developed a habit of documenting everything—journaling my process, reflecting on struggles. That's been invaluable. Future me can look back and see exactly how I got here.

What's Next

I'm leaving Meru with a promise kept. But this isn't the end—it's the beginning. I'm excited to explore data science, build real applications, and help others learn like I did.

The promise was simple. The journey changed everything.