DEV Community: The AI Pal

[Boost]

The AI Pal — Fri, 04 Sep 2026 17:23:22 +0000

The AI Pal

Sep 4

Run Qwen3-Coder-Next Locally on a Cost-Effective AI Home PC with llama.cpp

#ai #llm #local #tutorial

8 min read

Run Qwen3-Coder-Next Locally on a Cost-Effective AI Home PC with llama.cpp

The AI Pal — Fri, 04 Sep 2026 09:03:37 +0000

Why Try an MoE Model?
What MoE Means in Plain English
The Cost-Effective AI Home PC I Am Targeting
Install llama.cpp
Download a GGUF Model
Start the Model
Fit the Model to Your Hardware
Useful Local GenAI Projects
What to Expect
A Few Safety Checks
Final Thoughts

Running a local language model does not always mean buying a workstation with a huge graphics card.

Many cost-effective home PCs already have the parts needed for a useful local AI server: a reasonable CPU, plenty of system memory, an SSD, and a graphics card that can do some of the heavy lifting.

In my previous guide, I used Ollama because it makes the first local model easy to run. This time, I want to look at llama.cpp and a different kind of model: a Mixture-of-Experts, or MoE, model.

The goal is not to claim that an 8 GB graphics card can run every large model at full speed. It cannot. The goal is to show how a cost-effective AI home PC can host a surprisingly capable local model when we make sensible choices about model size, quantization, VRAM, and system RAM.

Why Try an MoE Model?

An MoE model can be interesting when you want better answers than a very small model but do not have the budget for a high-end AI GPU.

MoE models contain several smaller expert networks. For each part of a prompt, the model routes the work to only some of those experts. This means the model may have a large total parameter count while using a smaller number of active parameters for each token.

That can improve the quality-to-speed balance. It does not make the model free to run, though. The model still needs to store its weights somewhere, so system RAM and disk space remain important.

What MoE Means in Plain English

Think of a workshop with several specialists.

One person is good at code, another is good at writing, and another is good at reasoning. The manager does not ask everyone to work on every small task. It sends each task to the specialists most likely to help.

That is the basic idea behind a Mixture-of-Experts model.

There are two numbers worth keeping separate:

Total parameters: the size of all the experts and shared parts stored in the model
Active parameters: the smaller part used for a particular piece of text

The active number often helps explain why an MoE model can respond efficiently. The total number still matters when planning memory. A quantized model may fit partly in VRAM and partly in system RAM, but it still has to fit somewhere.

The Cost-Effective AI Home PC I Am Targeting

This guide is aimed at a fairly normal home PC rather than a dedicated server.

A comfortable starting point would be:

An NVIDIA or AMD GPU with 8 to 16 GB of VRAM
32 GB of system RAM
A modern six-core or eight-core CPU
An SSD with at least 50 GB of free space
Windows or Linux

You can try this with 16 GB of RAM, but 32 GB gives the operating system and the model more breathing room. If the model spills into system RAM, response speed will depend heavily on your CPU, memory bandwidth, and storage.

For the example in this post, I am using Qwen3-Coder-Next GGUF. It is designed for coding agents and has 80 billion total parameters, with about 3 billion active for each token. That is a useful example of why MoE models are interesting for local coding work.

There is an important catch. The model page lists the UD-Q4_K_M file at roughly 49 GB. An 8 GB GPU and 32 GB of RAM will not be a comfortable Q4 setup. For that machine, start with a smaller 2-bit quantization, or use a smaller MoE model. Treat Qwen3-Coder-Next Q4 as a stretch target for a PC with more than 45 GB of combined RAM and VRAM, plus extra room for the context window.

That means an 8 GB GPU will not hold the whole model by itself. That is fine. llama.cpp can divide the work between the GPU and CPU.

Install llama.cpp

The easiest Windows option is the official package available through WinGet:

winget install llama.cpp

Close and reopen PowerShell, then check that the command is available:

llama --version

You can also download a pre-built package from the llama.cpp releases page, or build it yourself from the llama.cpp repository.

The project requires models in the GGUF format. GGUF is simply the model file format used by llama.cpp and several other local AI tools.

Download a GGUF Model

The current llama.cpp command can download a compatible model directly from Hugging Face. This is convenient for a first test:

llama cli -hf unsloth/Qwen3-Coder-Next-GGUF:UD-IQ2_M

The first run downloads a large file, so do not be surprised if it takes a while. Make sure you have enough free space before starting.

If you prefer to download the file in a browser, save it somewhere simple, such as:

D:\models\Qwen3-Coder-Next-UD-IQ2_M.gguf

Use the exact filename you downloaded. Quantization names are part of the model choice. A Q4 file normally uses less memory than an 8-bit or 16-bit file, but it may give up a little quality.

Start the Model

For a quick terminal test, use:

llama cli -hf unsloth/Qwen3-Coder-Next-GGUF:UD-IQ2_M -c 4096

For a local service that other applications can use, start llama serve:

llama serve -hf unsloth/Qwen3-Coder-Next-GGUF:UD-IQ2_M -c 4096 --host 127.0.0.1 --port 8080

The server provides a browser interface and an OpenAI-compatible API on your own computer. The llama.cpp server guide explains the available options.

Some older Windows packages use the executable name llama-server.exe instead. The same idea looks like this when using a local file:

llama-server.exe `
  -m D:\models\Qwen3-Coder-Next-UD-IQ2_M.gguf `
  -c 4096 `
  --host 127.0.0.1 `
  --port 8080

Keeping the host set to 127.0.0.1 means the service is available only on your computer. That is a sensible default for a home lab.

Fit the Model to Your Hardware

The most important setting is not always the model name. It is how much of the model you ask the GPU to hold.

With a recent build of llama.cpp, the -ngl option controls GPU layer offloading. More layers on the GPU usually means better speed, but it also uses more VRAM.

For an 8 GB GPU and 32 GB of RAM, start conservatively:

llama serve `
  -hf unsloth/Qwen3-Coder-Next-GGUF:UD-IQ2_M `
  -c 4096 `
  -ngl 999 `
  --cpu-moe `
  --host 127.0.0.1 `
  --port 8080

The --cpu-moe option keeps the MoE expert weights in system RAM. The rest of the model can still use the GPU when there is room. This may be slower than keeping everything in VRAM, but it is a useful way to make a larger model work on a smaller card.

For Qwen3-Coder-Next, the model page recommends more than 30 GB of combined RAM and VRAM for its smaller 2-bit XL quantizations. That makes an 8 GB GPU and 32 GB of system RAM a realistic learning setup, although you should expect lower speed than a fully GPU-resident model. If you have 16 GB of VRAM and 32 GB of RAM, the Q4 files become a more realistic experiment, but still leave room for the operating system and context cache.

If the process runs out of memory, try these changes one at a time:

Lower the context size from 4096 to 2048
Use a smaller quantization or a smaller model
Keep more expert weights on the CPU
Close games, browsers, and other GPU-heavy applications
Use -ngl with a lower number instead of 999

If you have 12 or 16 GB of VRAM, try removing --cpu-moe and compare the speed and memory use. Every graphics card, driver, and model build behaves a little differently, so treat these commands as starting points rather than universal benchmarks.

Useful Local GenAI Projects

Once the server is running, it becomes more than a chat window. You can connect local applications to it and build small generative AI projects around your own data.

Here are a few practical home lab ideas:

Home Lab Assistant

Keep your Docker Compose files, network notes, and maintenance documentation in a private folder. Build a simple question-and-answer tool that helps you find commands and explains how your setup works.

Private Document Summaries

Use the local model to summarize manuals, project notes, or long text files without uploading them to a cloud service. Always check the summary against the original document.

Local Coding Helper

Point an editor or a small script at the OpenAI-compatible endpoint and use the model to explain code, suggest tests, or help understand an error. It may not replace a cloud coding model, but it is useful for private experiments.

Log File Companion

Send selected, sanitized log entries to the model and ask it to group repeated errors or explain what to investigate next. Avoid sending passwords, tokens, private keys, or personal information.

Personal Knowledge Search

The next step is retrieval-augmented generation, often called RAG. A small local program can search your notes first, then give the relevant passages to the model as context. This is a good project for learning how search, prompts, and local models fit together.

What to Expect

An MoE model on a cost-effective AI home PC can be useful, but it will not feel exactly like a hosted model running on professional hardware.

You may see a delay while the model loads. The first response can be slower than later responses. Long conversations use more memory because the model must keep more context available. If the model uses system RAM for expert weights, generation speed may drop noticeably.

That is not a failure. A local model is a trade-off between privacy, cost, speed, and quality. For short questions, coding explanations, document summaries, and home lab notes, the trade-off can be very reasonable.

Do not compare your tokens-per-second number directly with someone else's unless the model file, quantization, context size, prompt, backend, and hardware are also comparable.

A Few Safety Checks

Keep the server bound to 127.0.0.1 until you understand authentication and network access. Do not expose it directly to the public internet.

Be careful with tools that let a model run shell commands or change files. Start with read-only access and require confirmation before making changes.

Also remember that local does not mean automatically correct. A model can invent an answer with great confidence. Verify commands before running them, especially commands involving disks, firewalls, accounts, backups, or deletion.

Security note: Never send API keys, passwords, private keys, or confidential documents to a model unless you understand exactly where the data is going and how it is stored.

Final Thoughts

A cost-effective AI home PC can be a useful local AI machine even when its GPU is not designed for large language models.

The important part is to plan around the whole computer. VRAM matters, but so do system RAM, CPU speed, storage, cooling, and the model quantization. An MoE model can give you a larger model experience without requiring every parameter to be active for every token, while llama.cpp gives you control over how the workload is split between the GPU and CPU.

Start with a smaller context, keep the server private, and change one setting at a time. The best setup is not the one with the biggest model. It is the one that runs reliably and is useful enough that you keep using it.

In a future post, I will compare a few open MoE models on modest GPUs and look more closely at VRAM usage, CPU offloading, speed, and practical local generative AI workloads.

Running a Local LLM on an Older Computer: A Simple Home Lab Guide

The AI Pal — Thu, 03 Sep 2026 18:11:59 +0000

Why Run an AI Model at Home?
What Is a Local LLM?
What Kind of Computer Do You Need?
Installing Ollama
Downloading Your First Model
Starting a Conversation
What If the Model Is Too Slow?
Creating a Home Lab Assistant
Using Ollama from Another Program
Adding a Web Interface
Common Problems
Home Lab Project Ideas
Keep Your Setup Private
What Is Coming Next?
Final Thoughts

Why Run an AI Model at Home?

When most people hear "AI language model," they imagine expensive servers and powerful computers.

That is not always necessary.

You can run a small language model on an older laptop, a regular desktop, a mini PC, or a home lab server. It may not be as fast as the largest online AI tools, but it is still useful for learning and experimenting.

In this guide, I will show you how to run a local AI model using Ollama.

What Is a Local LLM?

LLM stands for Large Language Model. It is the technology used by many AI chat tools.

A local LLM runs directly on your computer instead of on a remote server owned by a company.

This has several advantages:

Your questions and files can stay on your computer
You can use it without paying for every request
It can work without an internet connection after the model is downloaded
You can experiment freely
It is a good way to learn how AI applications work

There are also some limitations. Smaller local models may not be as accurate or detailed as the largest cloud-based models. For a home lab, however, they are often good enough.

What Kind of Computer Do You Need?

You do not need a new gaming computer to get started.

A practical starting system might have:

8 GB of RAM for very small models
16 GB of RAM for a more comfortable experience
Four or more CPU cores
At least 10 GB of free storage
A graphics card is helpful, but not required

If you have an older computer, start with that. You can always move the setup to a more powerful machine later.

The amount of memory is usually more important than having the newest processor. Larger models need more memory, while smaller models can run on ordinary hardware.

Installing Ollama

Ollama is a tool that makes it easier to download and run local AI models.

Download and install it for your operating system. Ollama supports Windows, macOS, and Linux.

After installing it, open PowerShell, Terminal, or Command Prompt and run:

ollama --version

If you see a version number, Ollama is installed correctly.

You can also explore the Ollama GitHub repository if you want to learn more about how it works.

ollama / ollama

Get up and running with Kimi-K2.6, GLM-5.2, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models.

Ollama

Start building with open models.

Download

macOS

curl -fsSL https://ollama.com/install.sh | sh

or download manually

Windows

irm https://ollama.com/install.ps1 | iex

or download manually

Linux

curl -fsSL https://ollama.com/install.sh | sh

Manual install instructions

Docker

The official Ollama Docker image ollama/ollama is available on Docker Hub.

Libraries

Community

Get started

ollama

You'll be prompted to run a model or connect Ollama to your existing agents or applications such as Claude Code, OpenClaw, OpenCode , Codex, Copilot, and more.

Coding

To launch a specific integration:

ollama launch claude

Supported integrations include Claude Code, Codex, Copilot CLI, DeepSeek Harness, Droid, and OpenCode.

AI assistant

Use OpenClaw to turn Ollama into a personal AI assistant across WhatsApp, Telegram, Slack, Discord, and more:

ollama launch openclaw

Chat with a model

Run and chat with Gemma 4:

ollama run

…

View on GitHub

Downloading Your First Model

For an older or modest computer, start with a small model:

ollama pull llama3.2:3b

This downloads the model to your computer. The download may take a few minutes, depending on your internet connection. You only need to download it once.

The 3b in the model name means that it has around three billion parameters. You do not need to understand the technical details yet. The important thing to remember is that smaller models are easier for regular computers to run.

Starting a Conversation

After the model has finished downloading, start it with:

ollama run llama3.2:3b

You can now type questions directly into the terminal.

For example:

Explain Docker to me as if I have never used it before.

You can also ask the model to explain error messages, help write small scripts, or summarize text.

When you are finished, press Ctrl + D to exit.

That is all it takes to run your first local AI model.

What If the Model Is Too Slow?

If your computer becomes slow or the model takes a long time to respond, try a smaller model:

ollama pull phi3:mini
ollama run phi3:mini

Smaller models normally use less memory and respond faster. The trade-off is that their answers may be shorter or less detailed.

A simple way to think about it is:

Smaller models are faster and easier to run
Larger models usually give better answers but need more memory

There is no need to start with the largest model available. A small model is perfectly fine for learning.

Creating a Home Lab Assistant

Ollama lets you create a model with your own instructions.

Create a file named Modelfile and add this:

FROM llama3.2:3b

SYSTEM """
You are my home lab assistant.
Explain things clearly and avoid unnecessary technical language.
When giving commands, explain what each command does.
Warn me before suggesting commands that could delete or change data.
"""

Now create the custom model:

ollama create homelab-assistant -f Modelfile

Start it with:

ollama run homelab-assistant

You now have an assistant designed to help with home lab topics.

It can help explain Linux commands, Docker containers, networking concepts, Python scripts, and server errors. It will not always be correct, but it can be a useful learning companion.

Using Ollama from Another Program

Ollama also provides a local API. This allows your own scripts and applications to communicate with the model.

The API is usually available at:

http://localhost:11434

For example, you can send a request using curl from a Linux or macOS terminal:

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2:3b",
  "prompt": "Explain what a reverse proxy does",
  "stream": false
}'

This makes it possible to build small projects such as:

A private chatbot
A log file assistant
A documentation helper
A command-line question tool
A local coding assistant

You do not need to build an application immediately. It is enough to know that Ollama can connect to other programs when you are ready.

Adding a Web Interface

The terminal works well, but a web interface can be more comfortable.

Tools such as Open WebUI, LibreChat, and AnythingLLM can connect to Ollama and provide a browser-based chat interface.

For example, Open WebUI can be started with Docker:

docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:main

After it starts, open this address in your browser:

http://localhost:3000

Docker commands can vary slightly between operating systems. Check the Open WebUI documentation if the connection does not work immediately.

Common Problems

The Model Runs Slowly

This is normal when running a model using only the computer's processor.

Try closing other applications and switching to a smaller model. You may also get better performance by adding more RAM or using a supported graphics card.

For simple questions, slower responses are usually acceptable.

Your Computer Runs Out of Memory

This normally means that the model is too large for your system.

Try a smaller model and avoid running many other applications at the same time.

The Answers Are Incorrect

Local models can make mistakes. They may sound confident even when the information is wrong.

Be especially careful with answers about security, medical topics, legal matters, and commands that can delete or change data.

Security note: Use a local LLM as a learning assistant, but verify important information before acting on it. Be especially careful with commands that modify or delete files.

The Model Does Not Follow Instructions

Try giving it more information.

Instead of writing:

Fix this.

Try:

I am new to Linux. Explain why this command failed and show me a safe way to test the fix.

Clear prompts usually produce better answers.

Home Lab Project Ideas

Once everything is working, try building a small project.

Some ideas include:

A chatbot that explains server logs
A private assistant for your home lab documentation
A tool that summarizes text files
A script that explains Linux commands
A helper for learning Python
A question-and-answer tool for your personal notes

These projects do not need to be complicated. The goal is to understand how the different pieces work together.

Keep Your Setup Private

Even though the model runs locally, you should still be careful about how you expose it.

Do not make the Ollama API publicly available unless you understand the security risks. For a home lab, it is usually safest to keep it available only on your computer or local network.

It is also a good idea to:

Use a password for web interfaces
Keep your operating system updated
Keep Docker updated
Check unfamiliar commands before running them
Back up important configuration files

What Is Coming Next?

In my next post, I will explore how to run larger LLMs on a modest GPU with limited VRAM.

Having a smaller GPU does not necessarily limit you to tiny models. With compressed model formats, the right settings, and a few practical techniques, you can run surprisingly capable open models without investing in expensive hardware.

I will cover:

How to choose a model that fits your available VRAM
Practical ways to reduce memory usage
How to split the workload between GPU VRAM and system RAM
The balance between speed, memory usage, and answer quality
Some capable open models available for home lab use

The goal will remain the same: to keep the setup affordable, practical, and easy to follow.

If you want to get more from a modest GPU, follow me on DEV so you do not miss the next guide.

Final Thoughts

Running a local LLM is a useful and affordable home lab project.

You can start with an older computer, a small model, and Ollama. The experience may not be exactly the same as using a large online AI service, but that is part of the appeal.

You get to experiment, learn how local AI works, and keep your data close to home.

Start with a small model, try a few simple projects, and upgrade your hardware only when you understand what you actually need.

DEV Community: The AI Pal

[Boost]

Run Qwen3-Coder-Next Locally on a Cost-Effective AI Home PC with llama.cpp

Run Qwen3-Coder-Next Locally on a Cost-Effective AI Home PC with llama.cpp

Table of Contents

Why Try an MoE Model?

What MoE Means in Plain English

The Cost-Effective AI Home PC I Am Targeting

Install llama.cpp

Download a GGUF Model

Start the Model

Fit the Model to Your Hardware

Useful Local GenAI Projects

Home Lab Assistant

Private Document Summaries

Local Coding Helper

Log File Companion

Personal Knowledge Search

What to Expect

A Few Safety Checks

Final Thoughts

Running a Local LLM on an Older Computer: A Simple Home Lab Guide

Table of Contents

Why Run an AI Model at Home?

What Is a Local LLM?

What Kind of Computer Do You Need?

Installing Ollama

ollama / ollama

Get up and running with Kimi-K2.6, GLM-5.2, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models.

Ollama

Download

macOS

Windows

Linux

Docker

Libraries

Community

Get started

Coding

AI assistant

Chat with a model

Downloading Your First Model

Starting a Conversation

What If the Model Is Too Slow?

Creating a Home Lab Assistant

Using Ollama from Another Program

Adding a Web Interface

Common Problems

The Model Runs Slowly

Your Computer Runs Out of Memory

The Answers Are Incorrect

The Model Does Not Follow Instructions

Home Lab Project Ideas

Keep Your Setup Private

What Is Coming Next?

Final Thoughts