DEV Community: Genrius AI

How to Run a 4B LLM Locally on Windows with a Small NVIDIA GPU (3 Steps)

Genrius AI — Sat, 24 Oct 2026 15:54:53 +0000

Most guides to running a language model locally assume a big GPU or a long setup: Python, the CUDA Toolkit, a build step. This one does not. In three steps you will have Qwen3-4B, a 4-billion-parameter chat model, answering in your browser from a 1.24 GB file, on the kind of NVIDIA GPU that ships in an ordinary gaming laptop.

We use our own open-source build for this (Pingala-8, a patched llama.cpp), so we will also be clear about where it falls short.

Who this is for

You have a Windows 10 or 11 laptop or desktop.
It has an NVIDIA GeForce RTX 30, 40 or 50 series GPU.
You want a chat model that runs on your own machine, with no account, no cloud and nothing to compile.

What you need

A recent NVIDIA driver. That is the only thing to install. The zip already contains the CUDA runtime files (cudart64_12.dll, cublas64_12.dll, cublasLt64_12.dll), so you do not need the CUDA Toolkit.
GPU memory (VRAM). The Qwen3-4B model file is 1.24 GB; the Llama-3.1-8B file is 2.39 GB. The GPU also needs room for the chat context; see the tips below.
Disk space. The zip is 1.59 GB and unzips to 1.89 GB. The model downloads into the same folder: 1.24 GB for Qwen3-4B, 2.39 GB for Llama-3.1-8B. Plan for about 5 GB free for Qwen3-4B; you can delete the zip once it is unzipped.
Internet, once. The first run downloads the model from Hugging Face. Later runs start from the files on disk.

Step 1: Download the build

Open the v2.4-oneclick release page and, under Assets, download pingala8-windows-cuda12.zip (1.59 GB).

Step 2: Unzip and double-click the .bat

Right-click the zip, choose Extract All, and open the pingala8-windows-cuda12 folder. Inside are llama-server.exe and three other llama.cpp tools, the three CUDA DLLs, a README.txt, license files and three small scripts:

run_qwen3_4b.bat starts Qwen3-4B (about 1.2 GB download).
run_llama31_8b.bat starts Llama-3.1-8B (about 2.4 GB download).
env.bat holds settings that both scripts load; you do not run it yourself.

Double-click run_qwen3_4b.bat. A black console window opens. Here is what the script does, in order (it is 15 lines of plain text; open it in Notepad to check):

It moves into its own folder and creates a models folder there if there is none.
It downloads the two model parts, 4B-csq-vr3-00001-of-00002.gguf and 4B-csq-vr3-00002-of-00002.gguf, from the Genrius-AI/Qwen3-4B-Pingala8-2bit repository on Hugging Face, using the curl.exe built into Windows 10 and 11. A part that is already in models is skipped, which is why later starts are quick.
It runs env.bat, which sets eight GGML_... environment variables: the settings of our GPU kernels, the same values as in the paper's runs.
It opens http://127.0.0.1:8080 in your default browser.
It starts the server:

llama-server.exe -m models\4B-csq-vr3-00001-of-00002.gguf -ngl 99 -c 4096 --port 8080

-ngl 99 puts all of the model's layers on the GPU, -c 4096 sets a context of 4,096 tokens, and --port 8080 is the port the browser tab points at. Only the first part is passed; llama.cpp finds the second one by itself.

Windows may show a security prompt the first time, because the file came from the internet.

Step 3: Chat

The browser tab shows llama.cpp's built-in chat page, served by llama-server.exe on your own PC (127.0.0.1 means "this computer"). The script opens the browser a moment before the model has loaded, so if the page does not come up, wait a few seconds and refresh. Type a message and the reply streams in. On our test machine, an RTX 5070 Ti Laptop GPU, Qwen3-4B writes about 200 tokens per second.

The demo replays text and timing from a real run. Its chat window is recreated, so the llama.cpp page you get looks different.

By default llama-server listens only on 127.0.0.1, so other devices on your network cannot reach it. To stop, close the black window. To start again, double-click the same .bat; nothing is downloaded the second time.

Tips

4 GB of VRAM: use run_qwen3_4b.bat. The 1.24 GB file leaves room for the context and for what Windows itself is using.
8 GB of VRAM or more: both models fit, so you can also try run_llama31_8b.bat (2.39 GB, three files, in the same models folder).
These picks come from the file sizes. We measured on one laptop GPU and have not tried every card.
Close other GPU apps first. Games, video editors and busy browsers take VRAM. Task Manager, under Performance > GPU, shows "Dedicated GPU memory" in use.
Laptops: plug in. On battery, most laptop GPUs run at lower power and generate fewer tokens per second.

Troubleshooting

"Download failed. Check your internet connection and run this file again." The script prints this and waits for a key when a download fails. Fix the connection and double-click the .bat again.
The black window closes right away. The server stopped with an error, and the window closed with it. Type cmd in the folder's address bar to open a Command Prompt there, then type run_qwen3_4b.bat and press Enter: the window then stays open and you can read the message.
The model will not load after an interrupted download. The script skips any model file that already exists, even a half-finished one. Delete that model's .gguf files in the models folder and run the .bat again.
Errors about CUDA or the GPU. The build needs an NVIDIA RTX 30, 40 or 50 series GPU and a recent driver. Update the driver and try again.
The page cannot connect. Wait for the model to finish loading and refresh. If another program already uses port 8080, the server cannot start; close that program first.

Honest limits

NVIDIA and CUDA only. No AMD, Intel or Apple GPUs.
Long prompts are slower. Reading the prompt runs at 0.74x the speed of llama.cpp's IQ2_XXS format at the same file size, so a long pasted document means a longer wait before the first word. The script's 4,096-token context also limits how much fits in one chat.
One user at a time. Serving several chats at once is much slower; this is built for one person on one GPU.
Not in Ollama or LM Studio yet. The files need our patched llama.cpp. Stock llama.cpp, Ollama and LM Studio cannot load them, which is why the zip ships its own llama-server.exe.
Smaller is not free. On a five-task quiz, the 1.24 GB Qwen3-4B keeps 90% of the 8-bit model's score, so it will get more things wrong than the full-size model. We have not evaluated the safety behaviour of the compressed models.
One test GPU. Our speed numbers are from one RTX 5070 Ti Laptop GPU at batch 1, where it generates text 39% faster than llama.cpp's Q4_K_M at half its size. Your card will differ.

How it works, in one paragraph

The original Qwen3-4B is an 8.05 GB file of 16-bit weights; this one stores about 2 bits per weight, with no retraining. Before rounding, the weights are rotated so that no single value dominates; rounding errors are carried onto weights not yet rounded, so they tend to cancel; and the 2-bit codes follow an 8-state trellis, a small memory from one group of weights to the next that lets the decoder tell apart more values than the bits alone could address. A patched CUDA kernel decodes the codes inside each matrix multiply as the model runs. Our first article walks through each step with small worked examples: How We Shrank an AI Model by 85% and Kept 90% of Its Smarts.

How We Shrank an AI Model by 85% and Kept 90% of Its Smarts

Genrius AI — Sat, 24 Oct 2026 13:33:14 +0000

Qwen3-4B ships as an 8.05 GB file of 16-bit weights. In this post we walk through how a 2-bit quantizer gets the same model down to 1.24 GB (85% smaller) without retraining, and how to check whether the result is any good. Our worked example is Pingala-8, an open-source format we built on llama.cpp. Most of the ideas are not ours: the rotation and rounding come from QuIP and QuIP#, the trellis from QTIP and Ungerboeck's modem codes, and ikawrakow's IQ2_KT is its closest relative. We will also show which piece mattered less than we expected.

Why 2 bits is hard

At 2 bits per weight, each weight can take one of 4 values. Every weight is rounded to its nearest allowed value, and the difference is error. Two things make that error hurt.

It adds up. Each output of a layer is a sum of thousands of weight-times-input products. The rounding errors in those products do not cancel on their own, and they travel through dozens of layers.

Outliers. Weights are stored in small groups, each with a scale so its 4 levels cover the group's range. Take the group [3.0, 0.2, -0.1, 0.1]. To reach 3.0, the levels must be spread wide, say {-3, -1, 1, 3}. Now 0.2, -0.1 and 0.1 all land on -1 or +1, an error of about 1 each, larger than the weights themselves.

A good 2-bit quantizer must remove outliers, make errors cancel, and get more precision per bit. Those are the three tricks.

Trick 1: rotate the weights first

If R is an orthogonal matrix (R^T R = I), then W x = (W R^T)(R x). We can store the rotated weights W R^T, apply R on the input side, and the layer computes exactly the same result. A rotation keeps lengths but changes coordinates, so it can spread one large value across many coordinates.

The usual choice is a Hadamard matrix with random signs, because it is cheap to apply. The 4x4 version, scaled by 1/2:

      [ 1  1  1  1 ]
1/2 * [ 1 -1  1 -1 ]
      [ 1  1 -1 -1 ]
      [ 1 -1 -1  1 ]

Apply it to [3.0, 0.2, -0.1, 0.1] and you get [1.6, 1.3, 1.6, 1.5]. The sum of squares is 9.06 both times, but the largest value is now close to the others instead of 15 to 30 times bigger, and four evenly spaced levels fit it well. Real layers rotate many coordinates at once, and the rotated weights look roughly like a bell curve without extreme outliers.

Trick 2: round with error feedback

Plain rounding treats each weight on its own. The layer cares about the output, the sum of w_i * x_i.

Toy example: three weights [0.4, 0.4, 0.4] multiply inputs that are almost always equal, so the layer effectively uses their sum, 1.2. Rounding each to the nearest integer gives [0, 0, 0], off by 1.2. With feedback, the first 0.4 rounds to 0 and its error (0.4) is carried to the next weight, which becomes 0.8 and rounds to 1; that error (-0.2) is carried to the third, which becomes 0.2 and rounds to 0. The result [0, 1, 0] sums to 1, off by only 0.2.

LDLQ, from the QuIP paper, is the principled version. It quantizes the columns of a weight matrix one at a time and pushes each column's error onto the columns not yet quantized. How much goes where comes from H, the average of x x^T over inputs collected by running sample text through the model. If the inputs were uncorrelated, H would be diagonal and LDLQ would reduce to plain rounding.

Trick 3: trellis coding

IQ2_XXS, llama.cpp's smallest common 2-bit format, uses a codebook: each group of weights becomes the nearest entry of a fixed table, one group at a time. Trellis-coded quantization, used by QTIP and IQ2_KT, adds memory instead.

The encoder carries a small state from one group to the next. The state decides which subset (a "coset") of values the next group may use. The bits written for that group pick a value inside the subset and also set the next state. The decoder tracks the state for free, so the format can use twice as many values as the bits alone could address. The price: today's choice decides tomorrow's menu, so you cannot round each group on its own.

A toy with 1 bit per weight shows both sides. Four levels are split into two cosets: state 0 may use {-1.5, 0.5} and state 1 may use {-0.5, 1.5}. Bit 0 picks the lower value, bit 1 the higher, and the bit you wrote becomes the next state. Start in state 0 and encode [-0.4, 0.1, 1.5]:

Step	Target	Greedy: state, pick	Best path: state, pick
1	-0.4	0, 0.5	0, -1.5
2	0.1	1, -0.5	0, 0.5
3	1.5	0, 0.5	1, 1.5
Squared error		2.17	1.37

Greedy rounding takes the closest value at step 1 and is stuck with the wrong menus at steps 2 and 3. The best path accepts a worse first step to get the right menus later. Plain 1-bit rounding to {-1, 1}, with the same 3 bits, scores 1.42: greedy trellis rounding loses to it, the searched path beats it. On real data the advantage is an average over long sequences, not a guarantee at every step.

The search is the Viterbi algorithm: dynamic programming that keeps, for every state, the cheapest path ending there. Its cost grows linearly with length, and it runs once, offline, when the file is made.

Pingala-8 uses Ungerboeck's 8-state set-partitioned trellis over symbols of 4 weights, at about 2 bits per weight. Decoding is cheap (read bits, update state, look up values), and a fused CUDA kernel does it inside the matrix-vector product. Expensive search once, cheap decode on every token: that asymmetry is why trellis coding suits inference.

How we measured fairly

Quantization results are easy to flatter by accident. Our rules, which work for any comparison:

Equal file size, not equal "bits per weight". Formats count overhead differently (scales, embeddings, layers kept at higher precision). File bytes decide what fits in GPU memory, so we compared against files of the same size within a few megabytes: llama.cpp's IQ2_XXS and ik_llama.cpp's IQ2_KT.
Same harness for every file. WikiText-2 perplexity at context 4096 with the same tool and text windows. Five zero-shot quizzes (ARC-Easy, ARC-Challenge, HellaSwag, WinoGrande, MMLU) through one harness. Speed on one RTX 5070 Ti Laptop GPU, in one session.
Two kinds of quality. Perplexity measures how well the model predicts real text (lower is better). Quizzes check whether answers survive.
Paired comparisons. Every file scores the same windows and questions, so we compare item by item; shared difficulty cancels and error bars tighten.
Say what was tuned. The Qwen3-4B file is the last of many builds we checked on WikiText-2, so treat it as tuned. For Llama-3.1-8B and Qwen3-14B we wrote predictions down before running.

Results

Perplexity against file size. B4 is the released Pingala-8 file. Left: Qwen3-4B, where "post hoc" marks the tuned result. Right: Llama-3.1-8B scored without a start token in each window; the figures below add one.

Qwen3-4B, WikiText-2 perplexity at context 4096 (lower is better):

File	Size	Perplexity
Original, fp16	8.05 GB	12.46
llama.cpp IQ2_XXS	same as ours	28.61
ik_llama.cpp IQ2_KT	same as ours	18.15
Pingala-8	1.24 GB	16.64

On the five quizzes, Pingala-8 averages 50.8 against 41.9 for IQ2_XXS (+21%) and keeps 90% of the 8-bit Q8_0 average (56.3).

Speed for one user (batch 1), Qwen3-4B on the RTX 5070 Ti Laptop:

Measure	vs IQ2_XXS (same size)	vs Q4_K_M (twice the size)
Generating text	1.10x	1.39x
Reading the prompt	0.74x	-

Generating one token at a time mostly means reading every weight from memory, so fewer bytes give more tokens per second while decoding stays cheap. Reading a prompt processes many tokens at once and is limited by compute, which is where the extra decode work shows.

On Llama-3.1-8B (a 2.39 GB file), Pingala-8 scores 8.44 against 9.36 for IQ2_KT at the same size.

Where it loses

Qwen3-14B. IQ2_KT wins, 9.40 against our 9.57. We had predicted a win in writing before the run.
Prompt speed. 0.74x IQ2_XXS, so long prompts take longer before the first word. Serving several chats at once is also much slower; this is a one-user, one-GPU format.
CUDA only. It needs our patched llama.cpp and an NVIDIA GPU. Stock llama.cpp, Ollama and LM Studio cannot load the files.
Retrained models. Ternary Bonsai 4B scores 11.59 at 1.07 GB, smaller and better. It is a retrained model, not a compression of the original Qwen3-4B, so it took a training run. Post-training quantization is for models nobody has retrained for you.

What the ablation says

An ablation removes one part and measures the damage. We removed the trellis memory at equal bytes. The cost was 0.076 nats per token of extra loss, about 8% higher perplexity (e^0.076 is about 1.08).

Set it against our two gaps:

Against the tuned IQ2_KT file, itself a trellis format, the trellis memory accounts for most of our margin.
Against IQ2_XXS, it accounts for only 14% of our gain. The other 86% comes from the rotation, the LDLQ rounding and the rest of the pipeline.

The lesson for anyone building a 2-bit quantizer: rotation and error-feedback rounding carry most of the quality. Trellis memory explains most of the small margin between formats that are already close, and little of the large gap to a plain codebook format. We expected the trellis to be the main story; the ablation says otherwise, which is why it is worth running.

How to reproduce

Try it: download the one-click build (links below), unzip, and run run_qwen3_4b.bat on Windows or ./run.sh qwen on Linux. It downloads the model once and opens a local chat page. You need an NVIDIA RTX 30, 40 or 50 series GPU.
Rerun our numbers: the repository has the patch, the commands and the logs behind every number here.
Measure your own quant the same way with stock llama.cpp tools, on one machine, for files of the same size:

# perplexity: same text file and context give the same windows
./llama-perplexity -m model.gguf -f wiki.test.raw -c 4096 -ngl 99

# batch-1 speed: prompt processing (pp) and text generation (tg)
./llama-bench -m model.gguf -p 512 -n 128

Why the name

Pingala's Chandahsutra, an ancient Sanskrit work on poetic metre, lists the 8 patterns of three long or short syllables, the ganas. Our trellis has 8 states, so we named the format after him. The name is a credit; the trellis labels are Ungerboeck's.