Run Your Own Local AI Chat with OpenWebUI and llama.cpp - Windows

christian daniel — Sat, 28 Feb 2026 20:16:06 +0000

TL;DR: A local ChatGPT-like stack using OpenWebUI as the UI and llama.cpp as the inference server, with a GGUF model from Hugging Face. Everything talks over an OpenAI-compatible API. No API bills, no data leaving your machine.

Why this matters

Privacy: Prompts and replies stay on your machine.
No API bills: No usage-based pricing or quotas.
Control: You pick the model, quantization, and context size.
Open source: OpenWebUI and llama.cpp are free and auditable.

I wanted a local tool for LLM tasks that don't need a paid API: drafts, small scripts, experiments. This setup does that.

Who this is for

Anyone who wants a local AI chat without subscriptions. No prior LLM experience required; this is mostly wiring a UI to a local server.

My setup (Windows)

OS: Windows 11
RAM: 16 GB minimum; 32 GB helps for larger models
GPU: optional but recommended for speed (I have a GPU with 8 GB of VRAM)
Disk: Enough for multi-GB model files (often 4–8 GB per model)

A 7B model in Q4 quantization runs on many machines; bigger models need more memory.

Architecture overview

Three pieces:

OpenWebUI: the browser UI (chat, history, model selection)
llama.cpp server: local inference with an OpenAI-compatible HTTP API
GGUF model: weights you download once and keep on disk

OpenWebUI talks to llama-server over HTTP. No cloud in the loop.

Step 1: Install llama.cpp (Windows, prebuilt CUDA)

Prebuilt binaries are the fastest way to a working server.

1.1 Check your CUDA version (NVIDIA only)

In PowerShell:

nvidia-smi

Note the CUDA Version line (e.g. 12.x). You'll use it to choose the right llama.cpp build.

1.2 Download the release and CUDA runtime bundle

Open llama.cpp releases.
Pick the release that matches your CUDA version (e.g. CUDA 12) and download it.
Download the CUDA runtime DLL bundle from Assets (e.g. cudart-llama-bin-win-cuda-12).

The extra DLL bundle matters: the CUDA build often needs runtime DLLs that aren't on your PATH. Putting them next to the executables avoids "missing DLL" errors.

1.3 Extract and add to PATH

Extract the main archive to a stable folder, for example:

C:\Program Files\llama.cpp\

Add that folder to your system PATH (Windows search → Environment Variables → Path → Edit → New). That way you can run llama-server and llama-cli from any terminal without the full path.

1.4 Copy CUDA DLLs (NVIDIA only)

Extract the CUDA runtime bundle and copy all .dll files into the same folder as llama-server.exe (the one on your PATH).

1.5 Verify

Open a new terminal (so PATH is refreshed) and run:

llama-server --help

If you see help output, the install is good.

Step 2: Install and run OpenWebUI (Windows, no Docker)

OpenWebUI is a self-hosted chat UI. A straightforward option to install it is through a Python venv.

2.1 Create venv and install

python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install open-webui

Alternative (Conda):

conda create -n local_chat python=3.11 -y
conda activate local_chat
pip install open-webui

2.2 Run OpenWebUI

open-webui serve

Open http://localhost:8080 (or the port shown in the terminal). You'll see the UI; the model connection comes in Step 4.

Step 3: Download a GGUF model from Hugging Face and start the server

Start with a smaller model so you can confirm the pipeline before throwing RAM at bigger ones.

Example model used in this post: Qwen2.5-Coder-7B-Instruct-GGUF. I used the Q4_K_M quantized file.

On the Hugging Face repo you'll see several quantizations (Q2–Q8). Q4 is a good balance for local use: smaller file, decent quality.

3.1 Download the GGUF file

Download the Q4_K_M (or your chosen) .gguf file and put it in a stable folder, e.g.:

C:\Users\<YourUser>\.llm_models\

Replace <YourUser> with your Windows username.

3.2 Start the llama.cpp server

Use a port that doesn't clash with OpenWebUI (8080). Here we use 10000.

llama-server -m "C:\Users\<YourUser>\.llm_models\Qwen2.5-Coder-7B-Instruct-Q4_K_M.gguf" --port 10000

Leave this terminal open. You should now have an OpenAI-compatible API at:

http://localhost:10000

Step 4: Connect OpenWebUI to llama.cpp

With both the llama-server and OpenWebUI running:

Open OpenWebUI at http://localhost:8080.
Go to Settings → Connections (or Admin → Connections, depending on your OpenWebUI version).
Add an OpenAI-compatible connection (see screenshot below): Base URL http://localhost:10000/v1, API key empty or a placeholder like local if required.

Save, select the new connection/model in the UI, and send a test message. If the model answers, the stack is working.

What you get

A browser chat (OpenWebUI) talking to a model that runs on your machine (llama.cpp).
No external API calls.
No paid subscriptions.

Trade-offs and limitations

RAM/VRAM is the real limit. Bigger models and longer context need more memory.
Disk space adds up. Models live on disk (often several GB each, and you may keep several quantizations).
Smaller models have limits. On modest hardware, what you can run may not be enough for heavy reasoning, long-form planning, or high-stakes tasks.

Troubleshooting

OpenWebUI loads but no model appears

Confirm llama-server is running and that http://localhost:10000 responds (e.g. in a browser or with curl).
Make sure you didn’t use the same port as OpenWebUI (8080).

Connection fails

Try http://127.0.0.1:10000 instead of http://localhost:10000.
Check that Windows Firewall isn’t blocking local connections.

It’s slow

Use a smaller model or lower quantization (e.g. Q4).
Reduce context length if you increased it.
On NVIDIA: confirm you use the CUDA build and that the runtime DLLs are in the same folder as the executables.

Wrap-up

You now have a local chat stack: OpenWebUI for the UI, llama.cpp for inference, and a GGUF model from Hugging Face. Solid baseline for privacy-first, no-subscription experimentation. Next steps: try different models (general vs coder), other quantizations, or tuning context length for your workload.

DEV Community: christian daniel