DEV Community

Diogo Soares
Diogo Soares

Posted on Edited on

I Ran AI Offline for 7 Days. Was It Worth It?

#ai #llm #selfhosted #opensource

I typed "Olá" into my own offline chat and waited. Twenty-four seconds later, the model said hello back.

The model wasn't broken and the hardware wasn't dead. The slowness came from two settings I didn't know were on, and finding them taught me more about running AI offline than any benchmark chart.

Here's the setup. I bought a small mini PC, the NiPoGi P1, for €200, to run a few experiments at home. One of them: run two open models, Gemma 4 E4B for general questions and Qwen2.5-Coder 7B for code, on that box with no internet access at all, and use them from my Mac over the home network for a week.

This post covers:

  1. The hardware, and whether its "AMD Radeon" sticker means a real GPU.
  2. How to build the same setup, step by step: Ollama, the models, network access and a firewall that keeps the box off the internet.
  3. The real numbers, in tokens per second, on CPU and on the integrated GPU.
  4. Three things that made it slow that had nothing to do with the model.
  5. What I used it for, whether more RAM would help, and my verdict.

The hardware: a 15 W laptop chip in a box

The NiPoGi P1 is a mini PC that fits in one hand. Mine has this configuration:

Part What it is
CPU AMD Ryzen 3 3300U: 4 cores, 4 threads, Zen+, 15 W (specs)
GPU Radeon Vega 6, integrated in the processor
RAM 16 GB DDR4
Storage 512 GB SSD
Price €200, what I paid

Does it have a GPU? Yes, but not the kind people mean when they talk about running AI. The Vega 6 lives inside the processor and has no memory of its own: it borrows from the same 16 GB of system RAM. When I checked from inside the container, Vulkan reported it as AMD Radeon Graphics (RADV RAVEN), so it is usable. Keep the "shares the RAM" part in mind, because it explains most of the numbers later.

The 3300U is a 2019 laptop processor. I didn't buy this box to win benchmarks. I bought it to find out where the floor is: how much useful AI you get from cheap, quiet, low-power hardware.

Build the same setup, step by step

Any Linux machine on your home network works: an old laptop, a mini PC, a virtual machine. Mine runs Proxmox VE, so everything below runs inside a Debian 13 container with 4 cores and 10 GB of RAM, and the integrated GPU passed into it. Setting up Proxmox itself deserves its own article, so here I'll stick to what works on any Debian or Ubuntu box. Run the commands as root or with sudo.

Step 1: Install Ollama (the box still needs internet here)

Ollama downloads, runs and serves the models. The official installer sets it up as a systemd service with its own ollama user:

curl -fsSL https://ollama.com/install.sh | sh
ollama --version
Enter fullscreen mode Exit fullscreen mode

If piping a script into a shell makes you uneasy, read it first, or follow the manual install in the Ollama Linux docs. On my AMD box the installer warned that it couldn't detect an NVIDIA or AMD GPU. That warning is about CUDA and ROCm drivers, which this integrated GPU doesn't use anyway.

For the integrated GPU, install the Vulkan drivers and check that the GPU shows up:

apt install -y mesa-vulkan-drivers vulkan-tools
vulkaninfo --summary | grep deviceName
Enter fullscreen mode Exit fullscreen mode

Mine reported AMD Radeon Graphics (RADV RAVEN). This step is optional: everything works on the CPU alone.

Step 2: Download the models

Two downloads, done while the box still had internet:

ollama pull gemma4:e4b        # 6.6 GB on disk
ollama pull qwen2.5-coder:7b  # 4.7 GB on disk
Enter fullscreen mode Exit fullscreen mode

Gemma 4 E4B is Google's small multimodal model: "E4B" means about 4.5 billion effective parameters, around 8 billion counting embeddings (Ollama library). It reads images too, which matters later. Qwen2.5-Coder 7B is a code model from Alibaba's Qwen team.

Step 3: Open Ollama to the home network

By default Ollama only listens on localhost. A systemd override changes that, plus a few settings that matter on small hardware:

# /etc/systemd/system/ollama.service.d/override.conf
[Service]
Environment=OLLAMA_HOST=0.0.0.0:11434
Environment=OLLAMA_MAX_LOADED_MODELS=1
Environment=OLLAMA_NUM_PARALLEL=1
Environment=OLLAMA_KEEP_ALIVE=15m
Environment=OLLAMA_IGPU_ENABLE=1
Enter fullscreen mode Exit fullscreen mode 
systemctl daemon-reload
systemctl restart ollama
Enter fullscreen mode Exit fullscreen mode

Each line has a reason:

  • OLLAMA_HOST=0.0.0.0:11434 makes Ollama listen on the network, so the Mac can reach it.
  • OLLAMA_MAX_LOADED_MODELS=1: both models together don't fit in 10 GB, so only one stays in memory.
  • OLLAMA_NUM_PARALLEL=1: four slow cores serve one request at a time; parallel requests just queue.
  • OLLAMA_KEEP_ALIVE=15m unloads a model after 15 idle minutes.
  • OLLAMA_IGPU_ENABLE=1 lets Ollama use the integrated GPU through Vulkan. Without it, my Ollama version ignored the iGPU. Leave it out if you skipped the Vulkan step. The Vulkan backend itself has its own switch, OLLAMA_VULKAN; on my version (0.35.0) the server log already showed it on (OLLAMA_VULKAN:true), but on older versions you may need OLLAMA_VULKAN=1 as well.

Step 4: Test it from another machine

From the Mac (or any computer on the same network), ask the API which models it has. Replace the IP with your box's:

curl http://192.168.1.214:11434/api/tags
Enter fullscreen mode Exit fullscreen mode

If the JSON lists gemma4:e4b and qwen2.5-coder:7b, the network part works. The section "How I used it from the Mac" below turns this into a chat.

Step 5 (optional): A chat interface in the browser

Open WebUI gives you a ChatGPT-style page with history and image upload. I ran it in Docker on the same box:

docker run -d --name open-webui --restart unless-stopped --network host \
  -e OLLAMA_BASE_URL=http://127.0.0.1:11434 -e PORT=3000 \
  -v open-webui:/app/backend/data ghcr.io/open-webui/open-webui:main
Enter fullscreen mode Exit fullscreen mode

It opens on port 3000. :main follows the latest build; pin a release tag if you want the same version next month. Read the "Three things that made it slow" section before using it on slow hardware: two of its defaults cost me almost a minute per message.

What "offline" means in this setup

"Offline" here doesn't mean the cable is unplugged. The mini PC is on my home network, but it can't reach the internet. The models live on that box, and my Mac talks to them over the local network:

  Internet
     ✕   blocked by the firewall (nothing goes out)
     │
  Home router ── 192.168.1.0/24 (home network)
     │                     │
   Mac  ─── HTTP :11434 ──▶ Mini PC
  (terminal,  HTTP :3000 ──▶   Ollama + Gemma + Qwen
   browser)                    Open WebUI
Enter fullscreen mode Exit fullscreen mode

The Mac still has internet. What matters is where my prompts go: from the Mac, across the home network, to the mini PC, and nowhere else. The mini PC has no route out, so even a misbehaving tool inside it has nothing to send data to.

Step 6: Close the door

The internet was needed only for steps 1, 2 and 5. After that, a firewall that drops all outgoing traffic turns the box into an offline AI server. I did it with the Proxmox firewall on the container. On a plain Debian or Ubuntu box, ufw does the same:

ufw default deny incoming
ufw default deny outgoing
ufw allow from 192.168.1.0/24 to any port 22 proto tcp      # SSH
ufw allow from 192.168.1.0/24 to any port 11434 proto tcp   # Ollama API
ufw allow from 192.168.1.0/24 to any port 3000 proto tcp    # Open WebUI
ufw allow out 67/udp                                        # DHCP renewals
ufw enable
Enter fullscreen mode Exit fullscreen mode

What matters in it:

  • Outgoing traffic is denied by default, so the box can't open a connection to anything. Replies to the requests it accepts still go back, because the firewall tracks connections.
  • Inbound, only three ports, and only from the home network. Keep the SSH rule if the box has no screen, or you lock yourself out.
  • The DHCP rule lets the box ask to renew its IP lease; ufw's built-in rules already accept the router's reply. A DHCP reservation in your router keeps the address stable for the Mac. If you give the box a static IP instead, drop the rule. Networks with IPv6 or other extras may need more rules, so run the check below.
  • Docker writes its own firewall rules for ports published with -p, and traffic to those ports skips ufw's input rules. With --network host, as above, nothing is published: Open WebUI listens on the host itself, so the port 3000 rule applies. If you run other containers with -p, check them separately.

To check it, run something that needs the internet on the box and watch it fail:

curl -sS --max-time 5 https://ollama.com || echo "no internet, as expected"
Enter fullscreen mode Exit fullscreen mode

Updates and new models now mean opening the door on purpose (ufw disable, update, ufw enable). That's the point.

One honest limit: Ollama's API has no authentication. Anyone on my home network, including guests on the Wi-Fi, could use the models. The firewall keeps the internet out; it doesn't pick who inside the house gets in.

The numbers: tokens per second

I measured both models with the same coding prompt, generating 150 tokens, first on CPU only and then with Vulkan on the integrated GPU. This is one week of real use with a few measurements, not a controlled lab benchmark, so here is exactly how I measured:
Ollama 0.35.0, ollama run <model> --verbose
Models gemma4:e4b and qwen2.5-coder:7b, default 4-bit tags
Context Ollama's default (OLLAMA_CONTEXT_LENGTH not set)
Container 4 cores, 10 GB RAM, iGPU passed in
When The setup day, 2 October; not re-measured at the end of the week

Two numbers matter:

  • Generation: how fast the answer appears. This is the speed you feel.
  • Prompt reading: how fast the model reads what you send. It matters when you paste a long file or log.
Model Generation, CPU Generation, Vulkan Prompt reading, CPU Prompt reading, Vulkan
Gemma 4 E4B 5.5 tok/s 8.1 tok/s 14.4 tok/s 16.8 tok/s
Qwen2.5-Coder 7B 5.5 tok/s 4.5 tok/s 10.0 tok/s 27.7 tok/s

The integrated GPU didn't make everything faster. It changed which part was fast. For Qwen, prompt reading got almost three times faster while generation got slower. For Gemma, generation went up by about half. I don't have a clean explanation for why the two models moved in opposite directions, so I kept Vulkan on, because it won for the model I used the most.

The pattern makes sense once you remember the shared RAM. Reading a prompt is heavy math, and the GPU's extra compute helps. Generating each new token means reading the model's weights from memory again, and the iGPU reads from the same RAM, at the same speed, as the CPU.

To put 8 tokens per second in human terms: a long paragraph takes 20 to 30 seconds to finish appearing. You can read along, but you wait.

Switching models costs more. Only one fits in memory, so the first message after a switch spends 10 to 20 seconds loading the other one.

Three things that made it slow (none of them was the model)

Back to the 24-second "Olá". It came from two defaults I didn't know about. A third cost showed up later in the week.

1. Thinking mode was on by default

Gemma 4 supports a "thinking" mode, and in Ollama it's on by default. For a simple "Olá", the model wrote about 700 characters of internal reasoning before answering.

Mode Time to answer "Olá"
Thinking on (default) ~25 s
Thinking off ~2 s

The model wasn't slow; it was doing far more work than the question needed (192 generated tokens instead of 10). I turned thinking off by default (Open WebUI has a "Think" switch in the model's advanced parameters) and switch it on only for a hard question.

2. The chat interface added 2,000 tokens to every message

Even with thinking off, Open WebUI was still slow. The Ollama logs showed why: my "Olá" arrived at the model as 2,051 tokens of input, when the message itself is a handful.

Open WebUI attaches the descriptions of its built-in tools (memory, code interpreter, web search, image generation) to every message for models that support tool calling, and Gemma 4 does. On a cloud model you never notice. Here, the model spent about 55 seconds reading those tools and 1.5 seconds writing the answer. The tools were also useless: web search with no internet doesn't search anything.

The fix was to untick those capabilities in the model's settings (Admin Panel, Settings, Models) and to turn off the extra tasks that call the model behind your back: tag generation, follow-up suggestions and autocomplete. Every one of those is another request in a queue of one.

On slow hardware, the size of the prompt matters more than the size of the answer. Do the math for any tool you put in front of a local model: at 15 to 30 tokens per second of prompt reading, 10,000 tokens of system prompt means 5 to 11 minutes before the first word. That rules out most coding agents built for cloud models on this box.

3. The model switch

Not a bug, a cost: 10 to 20 seconds whenever I jumped between Gemma and Qwen. I learned to stay in one model per session.

How I used it from the Mac

Two ways, both over the home network.

The terminal. I installed only the Ollama client on the Mac and pointed it to the mini PC. I use the fish shell, so the setup is two lines plus one function per model:

set -Ux OLLAMA_HOST http://192.168.1.214:11434

function gemma
    ollama run gemma4:e4b --verbose --think=false $argv
end
Enter fullscreen mode Exit fullscreen mode

Typing gemma opens an interactive chat that runs on the mini PC. --verbose prints the speed after every answer, which turned the whole week into a small benchmark:

eval rate:            9.73 tokens/s
prompt eval rate:     7.54 tokens/s
total duration:       1.81s
Enter fullscreen mode Exit fullscreen mode

That run was a short greeting; a real three-sentence explanation of DNS took about 16 seconds end to end, at around 6 tokens per second.

The browser. Open WebUI on http://192.168.1.214:3000, for the two things the terminal does badly: keeping a searchable history and sending images.

During the week I used it for three things:

Use Model How it went
Quick questions ("what does this flag do?") Gemma, terminal The best fit. Short answers arrive in seconds.
Small code: explain a snippet, write a function, read a pasted log Qwen Useful for small pieces. Pasting code is where Vulkan's faster prompt reading pays off.
Reading screenshots and photos Gemma, Open WebUI It works, and nothing leaves the house. Slower than text.

What I didn't do is real daily work: long conversations, large files, an agent editing a project. At 5 to 8 tokens per second, the waiting adds up fast.

Would 32 GB of RAM make it faster?

This was my first instinct, and it's mostly wrong.

More RAM lets you load bigger models. It doesn't make generation faster. Generating each token means reading the model's weights from memory, so the ceiling is roughly memory bandwidth divided by model size. This is a rule of thumb, not an exact formula, but it's consistent with what I measured:

Dual-channel DDR4-2400:  2 channels × 8 bytes × 2,400 MT/s ≈ 38 GB/s (theoretical peak)
Qwen2.5-Coder 7B:        4.7 GB of weights

Ceiling ≈ 38 / 4.7 ≈ 8 tokens/s     (measured: 4.5 to 5.5)
Enter fullscreen mode Exit fullscreen mode

The 38 GB/s is the theoretical peak. Real memory traffic gets less than that, which is part of why the measured speed sits below the ceiling.

The 3300U's memory controller tops out at DDR4-2400 in dual channel (Notebookcheck). Going from 16 to 32 GB doesn't move that number. It lets you load a 14B model, which is about twice as big, so it would generate roughly half as fast.

One exception is worth checking. Reviews of this model show units shipping with a single 16 GB stick and one free slot. One stick means single channel, half the bandwidth. If that's your unit, two matched sticks are the one memory upgrade that can make it faster. Check before you buy:

sudo dmidecode -t memory | grep -E "Size|Locator|Speed"
Enter fullscreen mode Exit fullscreen mode

What would really change the experience is a different class of machine: one where the GPU has fast memory and lots of it. A mini PC like the Bosgame M5, with an AMD Ryzen AI Max+ 395 and 128 GB of LPDDR5X shared between CPU and GPU, has around 256 GB/s of bandwidth. By the same rule of thumb, that's a ceiling several times higher for the same model, and enough memory for much larger ones. I haven't tested one myself. A friend bought one, tested it and is happy with the results, but it's a big investment: at the time of writing it costs several times more than my mini PC, and prices move a lot with memory supply.

So, was it worth it?

My verdict after a week: worth it for quick questions, privacy and small pieces of text or code. Too slow for daily work.

Offline on the mini PC Cloud model
Privacy Prompts never leave the house Prompts go to a provider
Cost per use No API fees (hardware and electricity only) Subscription or per token
Works without internet Yes, anything on the home network No
Speed 5 to 8 tokens/s; long prompts are painful Fast enough not to think about it
Quality on hard problems A 4B to 7B model: fine for small tasks Much stronger
Agents and big contexts Not realistic here Yes

The part I'd keep even if I switched the box off tomorrow is what it taught me: on modest hardware, speed is mostly about memory bandwidth and prompt size, and the defaults of tools built for cloud models (thinking on, a dozen built-in tools) can make a working setup feel broken.

If you have an old laptop or a cheap mini PC, try this before buying anything. A weekend is enough to learn where your floor is, and the firewall rules above keep the experiment truly offline.

FAQ

Do I need Proxmox?
No. Proxmox only gives the models their own container on a box that runs other things too. The steps above work on any Debian or Ubuntu machine; I'll cover the Proxmox side in a separate article.

Why Gemma 4 E4B and Qwen2.5-Coder 7B?
They are small enough to fit in 10 GB of RAM at 4-bit quantization, and they cover two different needs: general questions with image input, and code.

Is Ollama's network API safe?
It has no authentication. Only expose it to a network you trust, and restrict it with firewall rules to the clients that need it.

Should I use the integrated GPU?
Measure both. On this box, Vulkan made prompt reading much faster for Qwen and generation faster for Gemma, but generation slower for Qwen. Ten minutes of testing with --verbose answers it for your hardware.

Takeaway

  • "Offline" can be a firewall rule. Block outbound traffic from the AI box and use it from your other devices over the home network.
  • An integrated GPU shares your RAM. It speeds up prompt reading more than generation.
  • Check the defaults first. Thinking mode and built-in tools turned a 2-second answer into 25 to 55 seconds.
  • Prompt size is the hidden cost. Every token a tool adds has to be read at 15 to 30 tokens per second.
  • More RAM buys capacity, not speed. Bandwidth sets the ceiling; dual channel matters more than 32 GB.
  • Local AI is a great second tool, not yet a replacement on cheap hardware.

What's the smallest machine you've run a local model on, and what did you actually use it for? Tell me in the comments.

I'm Diogo Soares, a Staff AI Engineer working on AI agents and legacy modernization. If this was useful, follow me here on DEV.to and on Instagram at @diogosoares.dev, where the numbers from this experiment become a carousel you can save for your own homelab. This article also gets a video walkthrough on my YouTube channel, Diogo Soares Dev, so subscribe there for the next ones. Let's also connect on LinkedIn, or check out diogosoares.dev.

Top comments (0)