<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Genrius AI</title>
    <description>The latest articles on DEV Community by Genrius AI (@genrius_ai).</description>
    <link>https://dev.to/genrius_ai</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4204750%2Fa09badfa-1b17-4a42-b21c-f435c600b215.png</url>
      <title>DEV Community: Genrius AI</title>
      <link>https://dev.to/genrius_ai</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/genrius_ai"/>
    <language>en</language>
    <item>
      <title>8 open models, 77-85% smaller: what 2-bit quantization costs and what it keeps</title>
      <dc:creator>Genrius AI</dc:creator>
      <pubDate>Sun, 01 Nov 2026 08:59:25 +0000</pubDate>
      <link>https://dev.to/genrius_ai/8-open-models-77-85-smaller-what-2-bit-quantization-costs-and-what-it-keeps-2112</link>
      <guid>https://dev.to/genrius_ai/8-open-models-77-85-smaller-what-2-bit-quantization-costs-and-what-it-keeps-2112</guid>
      <description>&lt;h2&gt;
  
  
  Try it
&lt;/h2&gt;

&lt;p&gt;Live chat with Qwen3.8-27B and Qwen3.5-2B v2 in your browser: &lt;a href="https://huggingface.co/spaces/harshit-genrius/pingala8-qwen35-live" rel="noopener noreferrer"&gt;https://huggingface.co/spaces/harshit-genrius/pingala8-qwen35-live&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Originally published at &lt;a href="https://genrius-ai.github.io/pingala8-llm/2-bit-quantization-8-models-cost.html" rel="noopener noreferrer"&gt;https://genrius-ai.github.io/pingala8-llm/2-bit-quantization-8-models-cost.html&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;This article was written with the help of AI; every number comes from our measured logs and the paper.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>machinelearning</category>
      <category>opensource</category>
    </item>
    <item>
      <title>Benchmarking 2-bit GGUFs fairly: paired windows, and the NaN trap on RTX 50 GPUs</title>
      <dc:creator>Genrius AI</dc:creator>
      <pubDate>Sun, 01 Nov 2026 08:59:23 +0000</pubDate>
      <link>https://dev.to/genrius_ai/benchmarking-2-bit-ggufs-fairly-paired-windows-and-the-nan-trap-on-rtx-50-gpus-1b5h</link>
      <guid>https://dev.to/genrius_ai/benchmarking-2-bit-ggufs-fairly-paired-windows-and-the-nan-trap-on-rtx-50-gpus-1b5h</guid>
      <description>&lt;h2&gt;
  
  
  Try it
&lt;/h2&gt;

&lt;p&gt;Live chat with Qwen3.8-27B and Qwen3.5-2B v2 in your browser: &lt;a href="https://huggingface.co/spaces/harshit-genrius/pingala8-qwen35-live" rel="noopener noreferrer"&gt;https://huggingface.co/spaces/harshit-genrius/pingala8-qwen35-live&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Originally published at &lt;a href="https://genrius-ai.github.io/pingala8-llm/benchmarking-2-bit-ggufs-paired-windows.html" rel="noopener noreferrer"&gt;https://genrius-ai.github.io/pingala8-llm/benchmarking-2-bit-ggufs-paired-windows.html&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;This article was written with the help of AI; every number comes from our measured logs and the paper.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>machinelearning</category>
      <category>opensource</category>
    </item>
    <item>
      <title>How we fit a 27B model on a 12 GB laptop GPU</title>
      <dc:creator>Genrius AI</dc:creator>
      <pubDate>Sun, 01 Nov 2026 08:25:14 +0000</pubDate>
      <link>https://dev.to/genrius_ai/how-we-fit-a-27b-model-on-a-12-gb-laptop-gpu-14eo</link>
      <guid>https://dev.to/genrius_ai/how-we-fit-a-27b-model-on-a-12-gb-laptop-gpu-14eo</guid>
      <description>&lt;h2&gt;
  
  
  In short
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Qwen3.8-27B goes from 53.8 GB of 16-bit weights to an 8.31 GB Pingala-8 file, 84.5% smaller (6.5x). It runs with every layer on one 12 GB RTX 5070 Ti Laptop GPU at 37.7 tokens/s (llama-bench, batch 1). &lt;/li&gt;
&lt;li&gt;Version 2 moves bits between tensor roles and layer depths, following the allocation in DASH-Q's published files. WikiText-2 perplexity fell from 7.34 to 6.89 at the same size, better on 20 of 20 test windows. &lt;/li&gt;
&lt;li&gt;DASH-Q IQ2_XS (8.55 GB) is still better than ours: 6.54, ahead on 20 of 20 windows. Against unsloth UD-IQ2_S (8.37 GB, 7.34) our lower average is not a significant win. &lt;/li&gt;
&lt;li&gt;The file needs our patched llama.cpp runtime on an NVIDIA GPU. Stock llama.cpp, Ollama and LM Studio cannot load it.
&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The problem: 53.8 GB of weights, 12 GB of memory
&lt;/h2&gt;

&lt;p&gt;Qwen3.8-27B is a 27-billion-parameter model. Its text weights, stored the way the Qwen team ships them at 16 bits each, come to 53.8 GB. That count leaves out the vision tower and the multi-token-prediction block, which we do not ship either. &lt;/p&gt;

&lt;p&gt;A model runs at full speed only when its weights sit in GPU memory, next to the cache that holds the conversation. A laptop GPU with 12 GB cannot hold 53.8 GB. Even at 4 bits per weight, the weights alone would be about 13.5 GB (a quarter of 53.8), more than the whole card before any cache. The usual answer is to leave some layers in system RAM, which slows generation to the speed of that slower memory. &lt;/p&gt;

&lt;p&gt;We wanted the whole model on the GPU. That means about 2 bits per weight. &lt;/p&gt;

&lt;h2&gt;
  
  
  What 2-bit means
&lt;/h2&gt;

&lt;p&gt;A weight is one number in one of the model's matrices. At 16 bits it takes 2 bytes. At 2 bits it takes a quarter of a byte, so on its own it could take only 4 different values. No useful format stores weights that crudely. 2-bit formats code small groups of weights together, keep a scale for each block of weights, and store a few sensitive tensors with more bits. The "2-bit" label describes the bulk of the file, not every byte of it. &lt;/p&gt;

&lt;p&gt;For our file the arithmetic is simple: 8.31 GB for 53.8 GB of 16-bit weights is 6.5x smaller, or about 2.5 bits per weight averaged over the whole file. The extra half bit pays for the scales, the token embedding and output head, and the tensors that get more than 2 bits. &lt;/p&gt;

&lt;p&gt;Fewer bits means rounding error, and the error adds up across 27 billion weights. We measure what that costs with WikiText-2 perplexity, a standard score for how well a model predicts real text; lower is better. Our &lt;a href="https://genrius-ai.github.io/pingala8-llm/2-bit-quantization-8-models-cost.html" rel="noopener noreferrer"&gt;article on 8 models&lt;/a&gt; explains the score in plain words. &lt;/p&gt;

&lt;p&gt;Pingala-8 codes each row of weights in groups of 4 with an 8-state trellis code, after a random Hadamard rotation and LDLQ rounding. These ideas come from QuIP, QuIP# and QTIP; the &lt;a href="https://genrius-ai.github.io/pingala8-llm/what-is-trellis-coded-quantization.html" rel="noopener noreferrer"&gt;trellis-coded quantization guide&lt;/a&gt; explains them. This article is about something else: how many bits each tensor gets. &lt;/p&gt;

&lt;h2&gt;
  
  
  The 8.31 GB file
&lt;/h2&gt;

&lt;p&gt;The release is 8,314,241,600 bytes in 9 parts on &lt;a href="https://huggingface.co/Genrius-AI/Qwen3.8-27B-Pingala8-2bit" rel="noopener noreferrer"&gt;Hugging Face&lt;/a&gt;. On our one test machine, an RTX 5070 Ti Laptop GPU with 12 GB under WSL2, it loads with every layer on the GPU (&lt;code&gt;-ngl 99&lt;/code&gt;) and leaves room for a 4,096-token context. The &lt;a href="https://genrius-ai.github.io/pingala8-llm/run-qwen3-8-27b-on-12gb-gpu.html" rel="noopener noreferrer"&gt;12 GB guide&lt;/a&gt; and the &lt;a href="https://genrius-ai.github.io/pingala8-llm/qwen3-8-27b-vram-requirements.html" rel="noopener noreferrer"&gt;VRAM requirements page&lt;/a&gt; show the memory arithmetic for longer contexts and other cards. We have not run this file on any other GPU. &lt;/p&gt;

&lt;h2&gt;
  
  
  The real-speed demo  [
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F2xmts0ntbhmc5h2vbi5m.gif" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F2xmts0ntbhmc5h2vbi5m.gif" alt="Terminal replay: Qwen3.8-27B Pingala-8 2-bit on one 12 GB RTX 5070 Ti Laptop GPU answers in 5 bullet points (feasibility, privacy, speed, cost, capability), 125 tokens, runtime eval 31.63 tokens per second." width="800" height="470"&gt;&lt;/a&gt;&lt;br&gt;
](&lt;a href="https://genrius-ai.github.io/pingala8-llm/media/demo27b.mp4" rel="noopener noreferrer"&gt;https://genrius-ai.github.io/pingala8-llm/media/demo27b.mp4&lt;/a&gt;) Real run, real speed. Text and timing captured from the run; terminal view re-rendered. Chat speed in this run: 31.63 tokens/s (runtime eval, 124 tokens, thinking off); the 37.7 tokens/s above is llama-bench tg64. &lt;a href="https://genrius-ai.github.io/pingala8-llm/media/demo27b.mp4" rel="noopener noreferrer"&gt;Watch the 1080p MP4&lt;/a&gt;.  &lt;/p&gt;

&lt;p&gt;The clip replays one chat run on that laptop: every token appears when it was captured, nothing is sped up, and only the terminal view is redrawn. It shows two speeds on purpose. 37.7 tokens/s is the llama-bench generation test. 31.63 tokens/s is the runtime's own count for this chat answer. They are different measurements, we have not yet looked into the gap, and we report both. &lt;/p&gt;
&lt;h2&gt;
  
  
  What we changed in version 2: bits by role and depth
&lt;/h2&gt;

&lt;p&gt;A transformer is a stack of layers, and each layer holds several kinds of matrix: the attention projections (&lt;code&gt;attn_qkv&lt;/code&gt; builds the queries, keys and values), the feed-forward matrices (&lt;code&gt;ffn_up&lt;/code&gt;, &lt;code&gt;ffn_gate&lt;/code&gt;, &lt;code&gt;ffn_down&lt;/code&gt;), and in Qwen3.8's linear-attention layers an output projection called &lt;code&gt;ssm_out&lt;/code&gt;. We call the kind of matrix its role, and its position in the stack its depth. A 2-bit file does not have to give every tensor the same number of bits. The question is where the extra bits do the most good. &lt;/p&gt;

&lt;p&gt;Version 1 copied the layout llama.cpp uses for its IQ2_XXS type. That layout spent 4.5 bits per weight on &lt;code&gt;attn_qkv&lt;/code&gt;, and the rest of the budget had to come from somewhere else. &lt;/p&gt;

&lt;p&gt;DASH-Q, a set of 2-bit GGUF files published by jkim96, splits the budget differently. We read the type of every tensor out of their Qwen3.8-27B file and built the same plan for ours: a bit budget for each role in each band of layers, from early to late. Same total bytes, different placement; the change was the allocation, not the coding method. &lt;/p&gt;

&lt;p&gt;The result, on the same 20 WikiText-2 windows:  &lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Version 1: perplexity 7.34. &lt;/li&gt;
&lt;li&gt;Version 2: perplexity 6.89, lower on all 20 windows, at the same 8.31 GB.
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The allocation idea is DASH-Q's, not ours. What we added is the evidence that the same plan helps a different coder. &lt;/p&gt;
&lt;h3&gt;
  
  
  The smaller model taught us the rest
&lt;/h3&gt;

&lt;p&gt;We then tried to copy the 27B plan onto Qwen3.5-2B. It did not help: perplexity 15.80 against 15.62 for version 1, better on only 22 of 70 windows. The 27B plan spent too much on the late feed-forward layers (4.1 bits per weight) and starved &lt;code&gt;ssm_out&lt;/code&gt; (2.6) for a model of this size. A plan learned on one model is not a plan for every model. &lt;/p&gt;

&lt;p&gt;The second gap was the embedding. In the 2B, the token embedding and the output head are the same matrix. DASH-Q stores it at Q4_K, 286 MB. We stored it at Q5_K, 350 MB. At equal file size, that left DASH-Q 489 MB for the other matrices against our 429 MB, 14% more. &lt;/p&gt;

&lt;p&gt;So the third try moved the embedding to Q4_K, which freed 61.6 MB, and used DASH-Q's own 2B plan for the rest. That file scored 14.06 at 778,189,408 bytes, slightly smaller than version 1 and better on 70 of 70 windows, a 10.0% drop in perplexity. It is now version 2 of our Qwen3.5-2B. The lesson we wrote down: the embedding and output head are part of the byte budget, so copy the leading file's whole-file allocation, not just its large matrices. &lt;/p&gt;
&lt;h2&gt;
  
  
  What failed: water-filling
&lt;/h2&gt;

&lt;p&gt;We also tried to derive an allocation ourselves instead of copying one. Water-filling is the textbook answer from rate-distortion theory: measure how much error each tensor suffers at each bit rate, then hand out bits where they cut the most error, until every tensor gains the same from one more bit. We used a per-tensor error measure from our encoder. &lt;/p&gt;

&lt;p&gt;We tested it where tests are cheap, on Qwen3.5-0.8B, at about 35 minutes of GPU time in all:     ********    &lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Qwen3.5-0.8B plan&lt;/th&gt;
&lt;th&gt;Perplexity&lt;/th&gt;
&lt;th&gt;vs DASH-Q plan&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;DASH-Q role and depth plan&lt;/td&gt;
&lt;td&gt;23.72&lt;/td&gt;
&lt;td&gt;reference&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Water-filling, variant 1&lt;/td&gt;
&lt;td&gt;26.72&lt;/td&gt;
&lt;td&gt;+0.120 nats/token&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Water-filling, variant 2&lt;/td&gt;
&lt;td&gt;25.71&lt;/td&gt;
&lt;td&gt;+0.081 nats/token&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;WikiText-2, lower is better. The last column is the paired per-window loss difference; positive means worse than the DASH-Q plan. &lt;/p&gt;

&lt;p&gt;Both variants lost clearly, so we stopped and logged the result. We do not know exactly why. One plausible reason is that the error of a tensor measured on its own is a poor guide to what the whole model loses when that tensor gets fewer bits. Either way, the DASH-Q plan won, and that is the one we shipped. &lt;/p&gt;
&lt;h2&gt;
  
  
  The honest comparison
&lt;/h2&gt;

&lt;p&gt;Here are the five Qwen3.8-27B files we measured on the same text, the same 20 windows and the same laptop. Lower is better.      ************     &lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;2-bit file&lt;/th&gt;
&lt;th&gt;Size&lt;/th&gt;
&lt;th&gt;PPL&lt;/th&gt;
&lt;th&gt;Against ours (v2)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;DASH-Q IQ2_XS&lt;/td&gt;
&lt;td&gt;8.55 GB&lt;/td&gt;
&lt;td&gt;6.54&lt;/td&gt;
&lt;td&gt;Better than ours on 20/20 windows&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Pingala-8 v2 (ours)&lt;/td&gt;
&lt;td&gt;8.31 GB&lt;/td&gt;
&lt;td&gt;6.89&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;unsloth UD-IQ2_S&lt;/td&gt;
&lt;td&gt;8.37 GB&lt;/td&gt;
&lt;td&gt;7.34&lt;/td&gt;
&lt;td&gt;Ours lower on average; not significant (9/20)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Pingala-8 v1 (ours)&lt;/td&gt;
&lt;td&gt;8.31 GB&lt;/td&gt;
&lt;td&gt;7.34&lt;/td&gt;
&lt;td&gt;v2 better on 20/20&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;llama.cpp IQ2_XXS, same imatrix&lt;/td&gt;
&lt;td&gt;8.43 GB&lt;/td&gt;
&lt;td&gt;7.88&lt;/td&gt;
&lt;td&gt;Ours lower&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;WikiText-2 test, 4,096-token windows, the first 20 windows for every file. Every window is in the &lt;a href="https://huggingface.co/datasets/Genrius-AI/2bit-gguf-wikitext2-eval" rel="noopener noreferrer"&gt;eval dataset&lt;/a&gt; and ranked on the &lt;a href="https://huggingface.co/spaces/Genrius-AI/2bit-gguf-leaderboard" rel="noopener noreferrer"&gt;2-bit GGUF leaderboard&lt;/a&gt;. &lt;/p&gt;

&lt;p&gt;DASH-Q is better. Its lead over us is 0.052 nats per token, with a 95% interval of plus or minus 0.012, and it holds on every window. That is about 5% in perplexity, for 0.24 GB more file. &lt;/p&gt;

&lt;p&gt;The unsloth comparison is the opposite case. Our average is lower, 6.89 against 7.34, but we are better on only 9 of 20 windows, and the paired interval (-0.064 plus or minus 0.075) includes zero. We do not count that as a win. Our &lt;a href="https://genrius-ai.github.io/pingala8-llm/benchmarking-2-bit-ggufs-paired-windows.html" rel="noopener noreferrer"&gt;benchmarking article&lt;/a&gt; explains the paired test and why a lower average is not enough. &lt;/p&gt;

&lt;p&gt;We did not compare speed with the rival files. On our GPU their IQ1_S, IQ2_S and IQ3_S tensors had to run on the CPU (the &lt;a href="https://genrius-ai.github.io/pingala8-llm/2bit-gguf-nan-perplexity-rtx-50.html" rel="noopener noreferrer"&gt;NaN page&lt;/a&gt; explains why), so a speed number from our machine would be unfair to them. Both rival files use standard llama.cpp types and load in stock llama.cpp; ours does not. &lt;/p&gt;

&lt;p&gt;So when is ours the right file? When you run our runtime on an NVIDIA card and want the smallest of the files that fit, and one measured fully on the GPU. If quality comes first, or you want stock llama.cpp, Ollama or LM Studio, take DASH-Q. &lt;/p&gt;
&lt;h2&gt;
  
  
  How to run it
&lt;/h2&gt;

&lt;p&gt;Build llama.cpp at commit &lt;code&gt;7ac59a6&lt;/code&gt; with the Pingala-8 CUDA patch and the Qwen3.5 loader change, then download the 9 parts and pass part 1:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;git clone https://github.com/Genrius-AI/pingala8-llm
git clone https://github.com/ggml-org/llama.cpp &amp;amp;&amp;amp; cd llama.cpp
git checkout 7ac59a6e3ad851cd41af00f678effab0598ba9a8
git apply ../pingala8-llm/patches/llama_cuda_vs_7ac59a6.patch
git apply ../pingala8-llm/patches/q35_loader_count_delta.patch
cmake -S . -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=120 -DCMAKE_BUILD_TYPE=Release -DLLAMA_OPENSSL=OFF
cmake --build build -j8 --target llama-cli llama-perplexity llama-bench

hf download Genrius-AI/Qwen3.8-27B-Pingala8-2bit --local-dir qwen3.8-27b-pingala8
export GGML_F44=4 GGML_F44_BATCH=2 GGML_VR_NATIVE=1 GGML_VRC_V=5 GGML_VRC_NWP=8 GGML_VR_RL=1 GGML_VR_PRMG=1 GGML_VR_PPC=2
./build/bin/llama-cli -m qwen3.8-27b-pingala8/Qwen3.8-27B-Pingala8-2bit-00001-of-00009.gguf -ngl 99 -c 4096
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The flag 120 matches our RTX 50 laptop GPU; use your card's value. This is the text model only, with no vision tower and no multi-token-prediction block, and it has not been tested with the one-click build. The &lt;a href="https://huggingface.co/Genrius-AI/Qwen3.8-27B-Pingala8-2bit#how-to-run" rel="noopener noreferrer"&gt;model card&lt;/a&gt; also has the exact perplexity command. &lt;/p&gt;

&lt;h2&gt;
  
  
  What we take from this
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Where the bits go mattered as much as how each weight is coded. The same coder gained 7.34 to 6.89 from a better plan. &lt;/li&gt;
&lt;li&gt;Copy the whole plan of the file that leads, including the embedding, and test it on the model you ship. &lt;/li&gt;
&lt;li&gt;Kill ideas on a small model first. Water-filling cost about 35 GPU minutes to rule out. &lt;/li&gt;
&lt;li&gt;DASH-Q still leads at this size. Closing that gap is the next job.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Try it
&lt;/h2&gt;

&lt;p&gt;Live chat with both models (Qwen3.8-27B and Qwen3.5-2B v2) in your browser: &lt;a href="https://huggingface.co/spaces/harshit-genrius/pingala8-qwen35-live" rel="noopener noreferrer"&gt;https://huggingface.co/spaces/harshit-genrius/pingala8-qwen35-live&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Originally published at &lt;a href="https://genrius-ai.github.io/pingala8-llm/how-we-fit-27b-model-on-12gb-laptop-gpu.html" rel="noopener noreferrer"&gt;https://genrius-ai.github.io/pingala8-llm/how-we-fit-27b-model-on-12gb-laptop-gpu.html&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;This article was written with the help of AI; every number comes from our measured logs and the paper.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>machinelearning</category>
      <category>opensource</category>
    </item>
    <item>
      <title>How to Compare Quantized LLMs Fairly in llama.cpp: Same Bytes, Same Windows, Same GPU</title>
      <dc:creator>Genrius AI</dc:creator>
      <pubDate>Sun, 25 Oct 2026 12:46:13 +0000</pubDate>
      <link>https://dev.to/genrius_ai/how-to-compare-quantized-llms-fairly-in-llamacpp-same-bytes-same-windows-same-gpu-240d</link>
      <guid>https://dev.to/genrius_ai/how-to-compare-quantized-llms-fairly-in-llamacpp-same-bytes-same-windows-same-gpu-240d</guid>
      <description>&lt;p&gt;Here are two GGUF files of Qwen3-4B, both sold as "2-bit". One scores a WikiText-2 perplexity of 28.61, the other 16.64. Is the second one better? Maybe. Before you believe a gap like that, you need three facts: how many bytes each file really takes, which text windows were scored and how, and whether the speed numbers came from one GPU in one session.&lt;/p&gt;

&lt;p&gt;This is a how-to for comparing quantized models in llama.cpp so that the numbers mean something. Every command below is stock llama.cpp and works on any GGUF file. Our worked example is Pingala-8, a 2-bit format we built (the two numbers above are llama.cpp's IQ2_XXS and ours). Its files need our patched llama.cpp build: stock llama.cpp, Ollama and LM Studio cannot load them. How the format works is in our &lt;a href="https://dev.to/genrius_ai/how-do-you-fit-a-4b-llm-in-124-gb-trellis-coded-2-bit-quantization-explained-with-real-benchmarks-3c9g"&gt;first article&lt;/a&gt;; this one is about measuring.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 1: Match file sizes in bytes, not labels
&lt;/h2&gt;

&lt;p&gt;A quant name describes a block format, not a file. &lt;code&gt;llama-quantize&lt;/code&gt; lists IQ2_XXS as "2.06 bpw", but an IQ2_XXS file stores the output layer as Q5_K, the token embeddings as Q2_K and, on models such as Qwen3 and Llama 3, the attention value tensors as Q4_K. The label tells you little about the file.&lt;/p&gt;

&lt;p&gt;The arithmetic for Qwen3-4B: the 16-bit file is 8.05 GB at 2 bytes per weight, so it holds about 4.0 billion weights. Our IQ2_XXS file is 1,246,621,312 bytes, about 2.5 bits per weight over the whole model, not 2.06. Pingala-8 is 1,243,153,792 bytes, also about 2.5.&lt;/p&gt;

&lt;p&gt;Why bytes? Bytes are what you download and what has to fit in VRAM, and they are the quality budget: a file 10% bigger can win simply by being bigger. GSQ's Q2_K of Qwen3-4B has a better perplexity than ours (14.37 against 16.64), but at 1.67 GB it is 34% larger, so that pair tells you little about either method.&lt;/p&gt;

&lt;p&gt;Measure on disk, and add up every part of a split file:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;stat&lt;/span&gt; &lt;span class="nt"&gt;-c&lt;/span&gt; &lt;span class="s1"&gt;'%s %n'&lt;/span&gt; model.gguf                  &lt;span class="c"&gt;# one file, exact bytes&lt;/span&gt;
&lt;span class="nb"&gt;du&lt;/span&gt; &lt;span class="nt"&gt;-cb&lt;/span&gt; qwen3-4b-pingala8/&lt;span class="k"&gt;*&lt;/span&gt;.gguf | &lt;span class="nb"&gt;tail&lt;/span&gt; &lt;span class="nt"&gt;-1&lt;/span&gt;   &lt;span class="c"&gt;# split GGUF: total of all parts&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two traps. Run any llama.cpp tool with &lt;code&gt;-lv 4&lt;/code&gt; (recent builds hide loader messages by default) and the loader prints &lt;code&gt;file size = ... GiB (... BPW)&lt;/code&gt;. That line counts tensor bytes only (no metadata) and uses GiB: 1,243,153,792 bytes is 1.24 GB on Hugging Face but 1.16 GiB in the log. Use &lt;code&gt;stat&lt;/code&gt; for matching sizes, and the loader's BPW, which includes the embeddings and output tensors, for bits per weight.&lt;/p&gt;

&lt;p&gt;Our rule: the candidate may not be larger than the file it is compared with. Pingala-8 is 3,467,520 bytes under IQ2_XXS and 2,760,960 bytes under the IQ2_KT file below.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 2: Same text, same context, same windows
&lt;/h2&gt;

&lt;p&gt;Perplexity depends on the text, the tokenizer, the window length and which tokens in each window are scored. Fix all four.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Same text, byte for byte.&lt;/strong&gt; Check the sha256 of the file you pass with &lt;code&gt;-f&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Same base model.&lt;/strong&gt; Windows are counted in tokens, so the same text gives 73 windows of 4,096 tokens on Qwen3-4B and 70 on Llama-3.1-8B. Perplexity is comparable only between files of one model.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Same mode.&lt;/strong&gt; &lt;code&gt;llama-perplexity&lt;/code&gt; has two, and they give different numbers:
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# default: every window starts with a BOS token; the second half of each window is scored&lt;/span&gt;
./llama-perplexity &lt;span class="nt"&gt;-m&lt;/span&gt; model.gguf &lt;span class="nt"&gt;-f&lt;/span&gt; wiki.test.raw &lt;span class="nt"&gt;-c&lt;/span&gt; 4096 &lt;span class="nt"&gt;-ngl&lt;/span&gt; 99

&lt;span class="c"&gt;# strided: every token is scored; windows after the first start mid-text&lt;/span&gt;
./llama-perplexity &lt;span class="nt"&gt;-m&lt;/span&gt; model.gguf &lt;span class="nt"&gt;-f&lt;/span&gt; wiki.test.raw &lt;span class="nt"&gt;-c&lt;/span&gt; 2049 &lt;span class="nt"&gt;--ppl-stride&lt;/span&gt; 4095 &lt;span class="nt"&gt;-ngl&lt;/span&gt; 99
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;-c 2049&lt;/code&gt; is not a typo. In strided mode the tool adds half the stride to &lt;code&gt;-c&lt;/code&gt;, and the log says so: "adjusting context size from 2049 to 4096".&lt;/p&gt;

&lt;p&gt;One pair of Llama-3.1-8B files under both modes (the IQ2_KT file was quantized from Q8_0):&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Llama-3.1-8B, WikiText-2&lt;/th&gt;
&lt;th&gt;Pingala-8&lt;/th&gt;
&lt;th&gt;IQ2_KT (ik_llama.cpp)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Default, BOS at every window&lt;/td&gt;
&lt;td&gt;8.44&lt;/td&gt;
&lt;td&gt;9.36&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Strided&lt;/td&gt;
&lt;td&gt;9.14&lt;/td&gt;
&lt;td&gt;10.04&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Same ranking, different numbers, because two things change at once. The default mode scores only the second half of each window, where every token has at least 2,048 tokens of context; strided mode also scores the poorly predicted early tokens. And in strided mode, windows after the first begin with an ordinary token: llama.cpp writes the BOS into the token list only after the window has been decoded. Models that use the BOS as an attention sink behave differently without it.&lt;/p&gt;

&lt;p&gt;That detail cost us a build. An earlier Llama-3.1-8B file of ours did well with a BOS and fell apart in strided mode, because every calibration sequence it had seen started with a BOS. The fix was to start every second calibration sequence mid-text. So compare with someone else's number only if the mode matches, and if your method calibrates on data, test both modes.&lt;/p&gt;

&lt;p&gt;One more check when formats need different runtimes. IQ2_KT lives in the ik_llama.cpp fork, so we ran a shared file through both binaries: our IQ2_XXS file scores 28.6083 in llama.cpp and 28.6037 in ik_llama.cpp. The runtimes are not the difference.&lt;/p&gt;

&lt;h3&gt;
  
  
  Compare per window, not two averages
&lt;/h3&gt;

&lt;p&gt;Both files score the same windows, so subtract window by window: shared difficulty cancels and the error bar shrinks. &lt;code&gt;llama-perplexity&lt;/code&gt; prints a running value after each window (&lt;code&gt;[1]...,[2]...&lt;/code&gt;), which is enough to recover each window's loss:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="c1"&gt;# paired.py A.log B.log: per-window NLL differences between two llama-perplexity runs
&lt;/span&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;math&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;statistics&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;st&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;sys&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;windows&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;ppl&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nf"&gt;float&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;findall&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;r&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;\[\d+\]([0-9.]+)&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;errors&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;ignore&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;read&lt;/span&gt;&lt;span class="p"&gt;())]&lt;/span&gt;
    &lt;span class="n"&gt;ln&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;math&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;ppl&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="c1"&gt;# running perplexity after k windows -&amp;gt; NLL of window k (every window has the same token count)
&lt;/span&gt;    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;ln&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]]&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="p"&gt;[(&lt;/span&gt;&lt;span class="n"&gt;k&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;ln&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;ln&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;k&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;range&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ln&lt;/span&gt;&lt;span class="p"&gt;))]&lt;/span&gt;

&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;windows&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;argv&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]),&lt;/span&gt; &lt;span class="nf"&gt;windows&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;argv&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
&lt;span class="n"&gt;d&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;y&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;y&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;zip&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="p"&gt;)]&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;A - B: &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;st&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;mean&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; +/- &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="mf"&gt;1.96&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;st&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;stdev&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;math&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sqrt&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; nats/token, &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
      &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;A better on &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="nf"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; of &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; windows&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;On our Qwen3-4B logs it gives -0.542 +/- 0.020 nats per token against IQ2_XXS, better on 73 of 73 windows. Adjacent windows are correlated, so treat the interval as optimistic when a margin is small.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 3: Use more than one quality metric
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Perplexity&lt;/strong&gt; says how well a file predicts real text. It is cheap and sensitive, but it is an average over tokens.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;KL divergence&lt;/strong&gt; says how far the quantized model's next-token distribution is from the original's, token by token (0 means identical). It needs a reference run that saves logits:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# 1. reference: save the logits of the fp16 (or Q8_0) model&lt;/span&gt;
./llama-perplexity &lt;span class="nt"&gt;-m&lt;/span&gt; qwen3-4b-q8_0.gguf &lt;span class="nt"&gt;-f&lt;/span&gt; wiki.test.raw &lt;span class="nt"&gt;-c&lt;/span&gt; 512 &lt;span class="nt"&gt;--chunks&lt;/span&gt; 72 &lt;span class="nt"&gt;-ngl&lt;/span&gt; 99 &lt;span class="nt"&gt;--kl-divergence-base&lt;/span&gt; q8.kld
&lt;span class="c"&gt;# 2. each quant: the tokens and context are read back from q8.kld&lt;/span&gt;
./llama-perplexity &lt;span class="nt"&gt;-m&lt;/span&gt; quant.gguf &lt;span class="nt"&gt;-c&lt;/span&gt; 512 &lt;span class="nt"&gt;-ngl&lt;/span&gt; 99 &lt;span class="nt"&gt;--kl-divergence-base&lt;/span&gt; q8.kld &lt;span class="nt"&gt;--kl-divergence&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The logits file is huge (the llama.cpp docs quote 37 GiB for Llama 3 on the full WikiText-2 test set), so cap &lt;code&gt;--chunks&lt;/code&gt;. If fp16 does not fit on your GPU, Q8_0 is a fair reference: on Qwen3-4B it scores 12.50 against 12.46 for fp16.&lt;/p&gt;

&lt;p&gt;The two can disagree. Of the IQ2_KT files of this size we built for Qwen3-4B, the one with the lowest perplexity (18.15) had the highest KL divergence: right on average, yet furthest from the original model. Report both.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Task scores&lt;/strong&gt; check whether answers survive. &lt;code&gt;llama-perplexity&lt;/code&gt; has zero-shot modes built in:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;./llama-perplexity &lt;span class="nt"&gt;-m&lt;/span&gt; model.gguf &lt;span class="nt"&gt;-c&lt;/span&gt; 2048 &lt;span class="nt"&gt;-ngl&lt;/span&gt; 99 &lt;span class="nt"&gt;--hellaswag&lt;/span&gt; &lt;span class="nt"&gt;--hellaswag-tasks&lt;/span&gt; 1000 &lt;span class="nt"&gt;-f&lt;/span&gt; hellaswag_val_full.txt
./llama-perplexity &lt;span class="nt"&gt;-m&lt;/span&gt; model.gguf &lt;span class="nt"&gt;-c&lt;/span&gt; 2048 &lt;span class="nt"&gt;-ngl&lt;/span&gt; 99 &lt;span class="nt"&gt;--winogrande&lt;/span&gt; &lt;span class="nt"&gt;-f&lt;/span&gt; winogrande-debiased-eval.csv
./llama-perplexity &lt;span class="nt"&gt;-m&lt;/span&gt; model.gguf &lt;span class="nt"&gt;-c&lt;/span&gt; 2048 &lt;span class="nt"&gt;-ngl&lt;/span&gt; 99 &lt;span class="nt"&gt;-np&lt;/span&gt; 5 &lt;span class="nt"&gt;--multiple-choice&lt;/span&gt; &lt;span class="nt"&gt;-f&lt;/span&gt; arc-easy-validation.bin
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;ARC questions can have five options, and our first ARC run aborted partway without &lt;code&gt;-np 5&lt;/code&gt;. Our five-task average on Qwen3-4B (ARC-Easy, ARC-Challenge, HellaSwag, WinoGrande, MMLU) is 50.8 for Pingala-8 against 41.9 for IQ2_XXS: 90% of the 8-bit Q8_0's 56.3, not 100%. A smaller file gets more answers wrong.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 4: Measure speed at batch 1, prompt and generation apart
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;llama-bench&lt;/code&gt; measures two different jobs. &lt;strong&gt;pp&lt;/strong&gt; (prompt processing, &lt;code&gt;-p 512 -n 0&lt;/code&gt;) reads many tokens at once and is limited by compute. &lt;strong&gt;tg&lt;/strong&gt; (token generation, &lt;code&gt;-p 0 -n 128&lt;/code&gt;) makes one token at a time and is limited by how fast the weights stream from memory. A smaller file usually helps tg; extra decode work in a complex format shows up in pp.&lt;/p&gt;

&lt;p&gt;Change nothing but the file: one GPU, one session, flash attention and KV cache type set explicitly, and a rotated order so that warm-up and clock drift do not always favour one file:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;models&lt;/span&gt;&lt;span class="o"&gt;=(&lt;/span&gt;iq2_xxs.gguf candidate.gguf q4_k_m.gguf&lt;span class="o"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;for &lt;/span&gt;r &lt;span class="k"&gt;in &lt;/span&gt;0 1 2&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;do
  for &lt;/span&gt;k &lt;span class="k"&gt;in &lt;/span&gt;0 1 2&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;do
    &lt;/span&gt;&lt;span class="nv"&gt;m&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;models&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="k"&gt;$((&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;k &lt;span class="o"&gt;+&lt;/span&gt; r&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;%&lt;/span&gt; &lt;span class="m"&gt;3&lt;/span&gt; &lt;span class="k"&gt;))&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;
    &lt;span class="k"&gt;for &lt;/span&gt;t &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="s2"&gt;"pp:-p 512 -n 0"&lt;/span&gt; &lt;span class="s2"&gt;"tg:-p 0 -n 128"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt;
      ./llama-bench &lt;span class="nt"&gt;-m&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$m&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="nt"&gt;-ngl&lt;/span&gt; 99 &lt;span class="nt"&gt;-fa&lt;/span&gt; 1 &lt;span class="nt"&gt;-ctk&lt;/span&gt; f16 &lt;span class="nt"&gt;-ctv&lt;/span&gt; f16 &lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;t&lt;/span&gt;&lt;span class="p"&gt;#*&lt;/span&gt;:&lt;span class="k"&gt;}&lt;/span&gt; &lt;span class="nt"&gt;-r&lt;/span&gt; 5 &lt;span class="nt"&gt;-o&lt;/span&gt; json &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;t&lt;/span&gt;&lt;span class="p"&gt;%%&lt;/span&gt;:&lt;span class="p"&gt;*&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;_r&lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;r&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;_&lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;m&lt;/span&gt;&lt;span class="p"&gt;%.gguf&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;.json"&lt;/span&gt;
    &lt;span class="k"&gt;done
  done
done&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Each JSON file holds &lt;code&gt;avg_ts&lt;/code&gt;, tokens per second. Divide it by the baseline's from the same round, then average the ratios over rounds.&lt;/p&gt;

&lt;p&gt;Our Qwen3-4B result on one RTX 5070 Ti Laptop GPU: about 200 tokens per second in generation, 39% faster than Q4_K_M (a file about twice the size), but prompt processing at 0.74x IQ2_XXS. Both belong in the same sentence. Batch size matters too: with 2 to 8 parallel sequences our file is much slower than IQ2_XXS, so benchmark the batch you will actually serve.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 5: Write the prediction down before the run
&lt;/h2&gt;

&lt;p&gt;If you tune on the test text, the final number is optimistic. Our Qwen3-4B file is the last of many builds we checked on WikiText-2, so we label it tuned and treat C4 as the only held-out text. For Llama-3.1-8B and Qwen3-14B we wrote a prediction and a decision rule into a timestamped file before the run. A template:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;prereg.txt       written before the run, never edited after
compare:    candidate vs IQ2_KT, no more bytes than IQ2_KT
metric:     WikiText-2, llama-perplexity strided, all windows
rule:       win only if the paired dNLL is below 0 and its 95% interval excludes 0
prediction: PPL &amp;lt;value&amp;gt; [&amp;lt;low&amp;gt;, &amp;lt;high&amp;gt;]
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Hash it (&lt;code&gt;sha256sum prereg.txt&lt;/code&gt;) and publish the hash with the result. Both of our predictions missed, in opposite directions: on Llama-3.1-8B the result beat our interval, and on Qwen3-14B we predicted a win and got a loss.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 6: Put the losses in the same table
&lt;/h2&gt;

&lt;p&gt;Qwen3-4B at equal size (WikiText-2, strided, 73 windows):&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;File&lt;/th&gt;
&lt;th&gt;Bytes&lt;/th&gt;
&lt;th&gt;Perplexity&lt;/th&gt;
&lt;th&gt;5-task quiz&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;fp16&lt;/td&gt;
&lt;td&gt;8.05 GB&lt;/td&gt;
&lt;td&gt;12.46&lt;/td&gt;
&lt;td&gt;-&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Q8_0&lt;/td&gt;
&lt;td&gt;-&lt;/td&gt;
&lt;td&gt;12.50&lt;/td&gt;
&lt;td&gt;56.3&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;IQ2_XXS (llama.cpp)&lt;/td&gt;
&lt;td&gt;1,246,621,312&lt;/td&gt;
&lt;td&gt;28.61&lt;/td&gt;
&lt;td&gt;41.9&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;IQ2_KT (ik_llama.cpp)&lt;/td&gt;
&lt;td&gt;1,245,914,752&lt;/td&gt;
&lt;td&gt;18.15&lt;/td&gt;
&lt;td&gt;46.5&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Pingala-8&lt;/td&gt;
&lt;td&gt;1,243,153,792&lt;/td&gt;
&lt;td&gt;16.64&lt;/td&gt;
&lt;td&gt;50.8&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;And next to it, the losses:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Qwen3-14B:&lt;/strong&gt; IQ2_KT wins, 9.40 against our 9.57, paired +0.018 +/- 0.008 nats per token, with our file better on only 12 of 73 windows. On held-out C4 the two tie (15.25 against 15.23).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Speed:&lt;/strong&gt; prompts at 0.74x and batched decoding at 0.18x to 0.34x IQ2_XXS.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Runtime:&lt;/strong&gt; a patched llama.cpp, CUDA only, measured on one GPU.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Retrained models:&lt;/strong&gt; Ternary Bonsai 4B scores 11.59 at 1.07 GB. It took a training run, so it belongs in a different comparison than post-training quantization.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The checklist
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;Sizes from &lt;code&gt;stat&lt;/code&gt;, in bytes, split parts added; the candidate no larger than the baseline.&lt;/li&gt;
&lt;li&gt;Bits per weight over the whole file, embeddings and output tensors included.&lt;/li&gt;
&lt;li&gt;Same text (sha256), same &lt;code&gt;-c&lt;/code&gt;, same perplexity mode, same window count; the mode stated next to every number.&lt;/li&gt;
&lt;li&gt;Perplexity compared only within one base model.&lt;/li&gt;
&lt;li&gt;A shared file through every runtime before comparing formats across runtimes.&lt;/li&gt;
&lt;li&gt;Paired per-window differences with an interval, not two averages.&lt;/li&gt;
&lt;li&gt;Perplexity, KL divergence against fp16 or Q8_0, and at least one task score.&lt;/li&gt;
&lt;li&gt;pp and tg reported apart, at the batch size you will use, on one GPU in one session, in rotated order, with &lt;code&gt;-fa&lt;/code&gt; and the KV cache type fixed.&lt;/li&gt;
&lt;li&gt;Prediction and decision rule written and hashed before the run; anything tuned on the test text labelled as tuned, with a held-out text.&lt;/li&gt;
&lt;li&gt;Losses in the same table as wins.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Links
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Code, patch and every log behind these numbers: &lt;a href="https://github.com/Genrius-AI/pingala8-llm" rel="noopener noreferrer"&gt;https://github.com/Genrius-AI/pingala8-llm&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Paper: &lt;a href="https://genrius-ai.github.io/pingala8-llm/paper.html" rel="noopener noreferrer"&gt;https://genrius-ai.github.io/pingala8-llm/paper.html&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Models: &lt;a href="https://huggingface.co/Genrius-AI/Qwen3-4B-Pingala8-2bit" rel="noopener noreferrer"&gt;https://huggingface.co/Genrius-AI/Qwen3-4B-Pingala8-2bit&lt;/a&gt; and &lt;a href="https://huggingface.co/Genrius-AI/Llama-3.1-8B-Pingala8-2bit" rel="noopener noreferrer"&gt;https://huggingface.co/Genrius-AI/Llama-3.1-8B-Pingala8-2bit&lt;/a&gt; (Built with Llama)&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Written by Harshit Goyal, founder of Genrius AI (ex-Google software engineer, IIIT Delhi alumnus).&lt;/p&gt;

&lt;p&gt;This article was written with the help of AI; every number comes from our measured logs and the paper.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>tutorial</category>
      <category>machinelearning</category>
      <category>llm</category>
    </item>
    <item>
      <title>How to Run a 4B LLM Locally on Windows with a Small NVIDIA GPU (3 Steps)</title>
      <dc:creator>Genrius AI</dc:creator>
      <pubDate>Sat, 24 Oct 2026 15:54:53 +0000</pubDate>
      <link>https://dev.to/genrius_ai/how-to-run-a-4b-llm-locally-on-windows-with-a-small-nvidia-gpu-3-steps-249k</link>
      <guid>https://dev.to/genrius_ai/how-to-run-a-4b-llm-locally-on-windows-with-a-small-nvidia-gpu-3-steps-249k</guid>
      <description>&lt;p&gt;Most guides to running a language model locally assume a big GPU or a long setup: Python, the CUDA Toolkit, a build step. This one does not. In three steps you will have Qwen3-4B, a 4-billion-parameter chat model, answering in your browser from a 1.24 GB file, on the kind of NVIDIA GPU that ships in an ordinary gaming laptop.&lt;/p&gt;

&lt;p&gt;We use our own open-source build for this (Pingala-8, a patched llama.cpp), so we will also be clear about where it falls short.&lt;/p&gt;

&lt;h2&gt;
  
  
  Who this is for
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;You have a Windows 10 or 11 laptop or desktop.&lt;/li&gt;
&lt;li&gt;It has an NVIDIA GeForce RTX 30, 40 or 50 series GPU.&lt;/li&gt;
&lt;li&gt;You want a chat model that runs on your own machine, with no account, no cloud and nothing to compile.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What you need
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;A recent NVIDIA driver.&lt;/strong&gt; That is the only thing to install. The zip already contains the CUDA runtime files (&lt;code&gt;cudart64_12.dll&lt;/code&gt;, &lt;code&gt;cublas64_12.dll&lt;/code&gt;, &lt;code&gt;cublasLt64_12.dll&lt;/code&gt;), so you do not need the CUDA Toolkit.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;GPU memory (VRAM).&lt;/strong&gt; The Qwen3-4B model file is 1.24 GB; the Llama-3.1-8B file is 2.39 GB. The GPU also needs room for the chat context; see the tips below.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Disk space.&lt;/strong&gt; The zip is 1.59 GB and unzips to 1.89 GB. The model downloads into the same folder: 1.24 GB for Qwen3-4B, 2.39 GB for Llama-3.1-8B. Plan for about 5 GB free for Qwen3-4B; you can delete the zip once it is unzipped.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Internet, once.&lt;/strong&gt; The first run downloads the model from Hugging Face. Later runs start from the files on disk.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Step 1: Download the build
&lt;/h2&gt;

&lt;p&gt;Open the &lt;a href="https://github.com/Genrius-AI/pingala8-llm/releases/latest" rel="noopener noreferrer"&gt;v2.4-oneclick release page&lt;/a&gt; and, under Assets, download &lt;code&gt;pingala8-windows-cuda12.zip&lt;/code&gt; (1.59 GB).&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 2: Unzip and double-click the .bat
&lt;/h2&gt;

&lt;p&gt;Right-click the zip, choose Extract All, and open the &lt;code&gt;pingala8-windows-cuda12&lt;/code&gt; folder. Inside are &lt;code&gt;llama-server.exe&lt;/code&gt; and three other llama.cpp tools, the three CUDA DLLs, a &lt;code&gt;README.txt&lt;/code&gt;, license files and three small scripts:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;run_qwen3_4b.bat&lt;/code&gt; starts Qwen3-4B (about 1.2 GB download).&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;run_llama31_8b.bat&lt;/code&gt; starts Llama-3.1-8B (about 2.4 GB download).&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;env.bat&lt;/code&gt; holds settings that both scripts load; you do not run it yourself.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Double-click &lt;code&gt;run_qwen3_4b.bat&lt;/code&gt;. A black console window opens. Here is what the script does, in order (it is 15 lines of plain text; open it in Notepad to check):&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;It moves into its own folder and creates a &lt;code&gt;models&lt;/code&gt; folder there if there is none.&lt;/li&gt;
&lt;li&gt;It downloads the two model parts, &lt;code&gt;4B-csq-vr3-00001-of-00002.gguf&lt;/code&gt; and &lt;code&gt;4B-csq-vr3-00002-of-00002.gguf&lt;/code&gt;, from the Genrius-AI/Qwen3-4B-Pingala8-2bit repository on Hugging Face, using the &lt;code&gt;curl.exe&lt;/code&gt; built into Windows 10 and 11. A part that is already in &lt;code&gt;models&lt;/code&gt; is skipped, which is why later starts are quick.&lt;/li&gt;
&lt;li&gt;It runs &lt;code&gt;env.bat&lt;/code&gt;, which sets eight &lt;code&gt;GGML_...&lt;/code&gt; environment variables: the settings of our GPU kernels, the same values as in the paper's runs.&lt;/li&gt;
&lt;li&gt;It opens &lt;code&gt;http://127.0.0.1:8080&lt;/code&gt; in your default browser.&lt;/li&gt;
&lt;li&gt;It starts the server:
&lt;/li&gt;
&lt;/ol&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;llama-server.exe -m models\4B-csq-vr3-00001-of-00002.gguf -ngl 99 -c 4096 --port 8080
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;-ngl 99&lt;/code&gt; puts all of the model's layers on the GPU, &lt;code&gt;-c 4096&lt;/code&gt; sets a context of 4,096 tokens, and &lt;code&gt;--port 8080&lt;/code&gt; is the port the browser tab points at. Only the first part is passed; llama.cpp finds the second one by itself.&lt;/p&gt;

&lt;p&gt;Windows may show a security prompt the first time, because the file came from the internet.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 3: Chat
&lt;/h2&gt;

&lt;p&gt;The browser tab shows llama.cpp's built-in chat page, served by &lt;code&gt;llama-server.exe&lt;/code&gt; on your own PC (127.0.0.1 means "this computer"). The script opens the browser a moment before the model has loaded, so if the page does not come up, wait a few seconds and refresh. Type a message and the reply streams in. On our test machine, an RTX 5070 Ti Laptop GPU, Qwen3-4B writes about 200 tokens per second.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fzv8mey61d7wxs16pv8xo.gif" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fzv8mey61d7wxs16pv8xo.gif" alt="Demo: Qwen3-4B as a 1.24 GB file replying on a laptop GPU" width="720" height="405"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The demo replays text and timing from a real run. Its chat window is recreated, so the llama.cpp page you get looks different.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;By default &lt;code&gt;llama-server&lt;/code&gt; listens only on 127.0.0.1, so other devices on your network cannot reach it. To stop, close the black window. To start again, double-click the same .bat; nothing is downloaded the second time.&lt;/p&gt;

&lt;h2&gt;
  
  
  Tips
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;4 GB of VRAM:&lt;/strong&gt; use &lt;code&gt;run_qwen3_4b.bat&lt;/code&gt;. The 1.24 GB file leaves room for the context and for what Windows itself is using.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;8 GB of VRAM or more:&lt;/strong&gt; both models fit, so you can also try &lt;code&gt;run_llama31_8b.bat&lt;/code&gt; (2.39 GB, three files, in the same &lt;code&gt;models&lt;/code&gt; folder).&lt;/li&gt;
&lt;li&gt;These picks come from the file sizes. We measured on one laptop GPU and have not tried every card.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Close other GPU apps first.&lt;/strong&gt; Games, video editors and busy browsers take VRAM. Task Manager, under Performance &amp;gt; GPU, shows "Dedicated GPU memory" in use.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Laptops: plug in.&lt;/strong&gt; On battery, most laptop GPUs run at lower power and generate fewer tokens per second.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Troubleshooting
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;"Download failed. Check your internet connection and run this file again."&lt;/strong&gt; The script prints this and waits for a key when a download fails. Fix the connection and double-click the .bat again.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The black window closes right away.&lt;/strong&gt; The server stopped with an error, and the window closed with it. Type &lt;code&gt;cmd&lt;/code&gt; in the folder's address bar to open a Command Prompt there, then type &lt;code&gt;run_qwen3_4b.bat&lt;/code&gt; and press Enter: the window then stays open and you can read the message.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The model will not load after an interrupted download.&lt;/strong&gt; Since v2.4.1 the script checks each model file's size and checksum and downloads a broken file again on its own. On the older v2.4 build, delete that model's &lt;code&gt;.gguf&lt;/code&gt; files in the &lt;code&gt;models&lt;/code&gt; folder and run the .bat again.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Errors about CUDA or the GPU.&lt;/strong&gt; The build needs an NVIDIA RTX 30, 40 or 50 series GPU and a recent driver. Update the driver and try again.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The page cannot connect.&lt;/strong&gt; Wait for the model to finish loading and refresh. If another program already uses port 8080, the server cannot start; close that program first.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Honest limits
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;NVIDIA and CUDA only.&lt;/strong&gt; No AMD, Intel or Apple GPUs.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Long prompts are slower.&lt;/strong&gt; Reading the prompt runs at 0.74x the speed of llama.cpp's IQ2_XXS format at the same file size, so a long pasted document means a longer wait before the first word. The script's 4,096-token context also limits how much fits in one chat.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;One user at a time.&lt;/strong&gt; Serving several chats at once is much slower; this is built for one person on one GPU.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Not in Ollama or LM Studio yet.&lt;/strong&gt; The files need our patched llama.cpp. Stock llama.cpp, Ollama and LM Studio cannot load them, which is why the zip ships its own &lt;code&gt;llama-server.exe&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Smaller is not free.&lt;/strong&gt; On a five-task quiz, the 1.24 GB Qwen3-4B keeps 90% of the 8-bit model's score, so it will get more things wrong than the full-size model. We have not evaluated the safety behaviour of the compressed models.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;One test GPU.&lt;/strong&gt; Our speed numbers are from one RTX 5070 Ti Laptop GPU at batch 1, where it generates text 39% faster than llama.cpp's Q4_K_M at half its size. Your card will differ.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  How it works, in one paragraph
&lt;/h2&gt;

&lt;p&gt;The original Qwen3-4B is an 8.05 GB file of 16-bit weights; this one stores about 2 bits per weight, with no retraining. Before rounding, the weights are rotated so that no single value dominates; rounding errors are carried onto weights not yet rounded, so they tend to cancel; and the 2-bit codes follow an 8-state trellis, a small memory from one group of weights to the next that lets the decoder tell apart more values than the bits alone could address. A patched CUDA kernel decodes the codes inside each matrix multiply as the model runs. Our first article walks through each step with small worked examples: &lt;a href="https://dev.to/genrius_ai/how-do-you-fit-a-4b-llm-in-124-gb-trellis-coded-2-bit-quantization-explained-with-real-benchmarks-3c9g"&gt;How We Shrank an AI Model by 85% and Kept 90% of Its Smarts&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Links
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;One-click build (Windows and Linux): &lt;a href="https://github.com/Genrius-AI/pingala8-llm/releases/latest" rel="noopener noreferrer"&gt;https://github.com/Genrius-AI/pingala8-llm/releases/latest&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Code, patches and logs: &lt;a href="https://github.com/Genrius-AI/pingala8-llm" rel="noopener noreferrer"&gt;https://github.com/Genrius-AI/pingala8-llm&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Models on Hugging Face: &lt;a href="https://huggingface.co/Genrius-AI/Qwen3-4B-Pingala8-2bit" rel="noopener noreferrer"&gt;https://huggingface.co/Genrius-AI/Qwen3-4B-Pingala8-2bit&lt;/a&gt; and &lt;a href="https://huggingface.co/Genrius-AI/Llama-3.1-8B-Pingala8-2bit" rel="noopener noreferrer"&gt;https://huggingface.co/Genrius-AI/Llama-3.1-8B-Pingala8-2bit&lt;/a&gt; (Built with Llama; the Llama 3.1 Community License applies)&lt;/li&gt;
&lt;li&gt;Paper (DOI): &lt;a href="https://doi.org/10.5281/zenodo.23481757" rel="noopener noreferrer"&gt;https://doi.org/10.5281/zenodo.23481757&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;On Linux: extract &lt;code&gt;pingala8-linux-cuda13.tar.gz&lt;/code&gt;, run &lt;code&gt;./run.sh qwen&lt;/code&gt; (or &lt;code&gt;./run.sh llama&lt;/code&gt;), then open &lt;code&gt;http://127.0.0.1:8080&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Written by Harshit Goyal, founder of Genrius AI (ex-Google software engineer, IIIT Delhi alumnus).&lt;/p&gt;

&lt;p&gt;&lt;em&gt;This article was written with the help of AI; every number comes from our measured logs and the paper.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>tutorial</category>
      <category>beginners</category>
    </item>
    <item>
      <title>How We Shrank an AI Model by 85% and Kept 90% of Its Smarts</title>
      <dc:creator>Genrius AI</dc:creator>
      <pubDate>Sat, 24 Oct 2026 13:33:14 +0000</pubDate>
      <link>https://dev.to/genrius_ai/how-do-you-fit-a-4b-llm-in-124-gb-trellis-coded-2-bit-quantization-explained-with-real-benchmarks-3c9g</link>
      <guid>https://dev.to/genrius_ai/how-do-you-fit-a-4b-llm-in-124-gb-trellis-coded-2-bit-quantization-explained-with-real-benchmarks-3c9g</guid>
      <description>&lt;p&gt;Qwen3-4B ships as an 8.05 GB file of 16-bit weights. In this post we walk through how a 2-bit quantizer gets the same model down to 1.24 GB (85% smaller) without retraining, and how to check whether the result is any good. Our worked example is Pingala-8, an open-source format we built on llama.cpp. Most of the ideas are not ours: the rotation and rounding come from QuIP and QuIP#, the trellis from QTIP and Ungerboeck's modem codes, and ikawrakow's IQ2_KT is its closest relative. We will also show which piece mattered less than we expected.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why 2 bits is hard
&lt;/h2&gt;

&lt;p&gt;At 2 bits per weight, each weight can take one of 4 values. Every weight is rounded to its nearest allowed value, and the difference is error. Two things make that error hurt.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;It adds up.&lt;/strong&gt; Each output of a layer is a sum of thousands of weight-times-input products. The rounding errors in those products do not cancel on their own, and they travel through dozens of layers.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Outliers.&lt;/strong&gt; Weights are stored in small groups, each with a scale so its 4 levels cover the group's range. Take the group [3.0, 0.2, -0.1, 0.1]. To reach 3.0, the levels must be spread wide, say {-3, -1, 1, 3}. Now 0.2, -0.1 and 0.1 all land on -1 or +1, an error of about 1 each, larger than the weights themselves.&lt;/p&gt;

&lt;p&gt;A good 2-bit quantizer must remove outliers, make errors cancel, and get more precision per bit. Those are the three tricks.&lt;/p&gt;

&lt;h2&gt;
  
  
  Trick 1: rotate the weights first
&lt;/h2&gt;

&lt;p&gt;If R is an orthogonal matrix (R^T R = I), then W x = (W R^T)(R x). We can store the rotated weights W R^T, apply R on the input side, and the layer computes exactly the same result. A rotation keeps lengths but changes coordinates, so it can spread one large value across many coordinates.&lt;/p&gt;

&lt;p&gt;The usual choice is a Hadamard matrix with random signs, because it is cheap to apply. The 4x4 version, scaled by 1/2:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;      [ 1  1  1  1 ]
1/2 * [ 1 -1  1 -1 ]
      [ 1  1 -1 -1 ]
      [ 1 -1 -1  1 ]
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Apply it to [3.0, 0.2, -0.1, 0.1] and you get [1.6, 1.3, 1.6, 1.5]. The sum of squares is 9.06 both times, but the largest value is now close to the others instead of 15 to 30 times bigger, and four evenly spaced levels fit it well. Real layers rotate many coordinates at once, and the rotated weights look roughly like a bell curve without extreme outliers.&lt;/p&gt;

&lt;h2&gt;
  
  
  Trick 2: round with error feedback
&lt;/h2&gt;

&lt;p&gt;Plain rounding treats each weight on its own. The layer cares about the output, the sum of w_i * x_i.&lt;/p&gt;

&lt;p&gt;Toy example: three weights [0.4, 0.4, 0.4] multiply inputs that are almost always equal, so the layer effectively uses their sum, 1.2. Rounding each to the nearest integer gives [0, 0, 0], off by 1.2. With feedback, the first 0.4 rounds to 0 and its error (0.4) is carried to the next weight, which becomes 0.8 and rounds to 1; that error (-0.2) is carried to the third, which becomes 0.2 and rounds to 0. The result [0, 1, 0] sums to 1, off by only 0.2.&lt;/p&gt;

&lt;p&gt;LDLQ, from the QuIP paper, is the principled version. It quantizes the columns of a weight matrix one at a time and pushes each column's error onto the columns not yet quantized. How much goes where comes from H, the average of x x^T over inputs collected by running sample text through the model. If the inputs were uncorrelated, H would be diagonal and LDLQ would reduce to plain rounding.&lt;/p&gt;

&lt;h2&gt;
  
  
  Trick 3: trellis coding
&lt;/h2&gt;

&lt;p&gt;IQ2_XXS, llama.cpp's smallest common 2-bit format, uses a codebook: each group of weights becomes the nearest entry of a fixed table, one group at a time. Trellis-coded quantization, used by QTIP and IQ2_KT, adds memory instead.&lt;/p&gt;

&lt;p&gt;The encoder carries a small state from one group to the next. The state decides which subset (a "coset") of values the next group may use. The bits written for that group pick a value inside the subset and also set the next state. The decoder tracks the state for free, so the format can use twice as many values as the bits alone could address. The price: today's choice decides tomorrow's menu, so you cannot round each group on its own.&lt;/p&gt;

&lt;p&gt;A toy with 1 bit per weight shows both sides. Four levels are split into two cosets: state 0 may use {-1.5, 0.5} and state 1 may use {-0.5, 1.5}. Bit 0 picks the lower value, bit 1 the higher, and the bit you wrote becomes the next state. Start in state 0 and encode [-0.4, 0.1, 1.5]:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Step&lt;/th&gt;
&lt;th&gt;Target&lt;/th&gt;
&lt;th&gt;Greedy: state, pick&lt;/th&gt;
&lt;th&gt;Best path: state, pick&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;-0.4&lt;/td&gt;
&lt;td&gt;0, 0.5&lt;/td&gt;
&lt;td&gt;0, -1.5&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;0.1&lt;/td&gt;
&lt;td&gt;1, -0.5&lt;/td&gt;
&lt;td&gt;0, 0.5&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;td&gt;1.5&lt;/td&gt;
&lt;td&gt;0, 0.5&lt;/td&gt;
&lt;td&gt;1, 1.5&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Squared error&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;2.17&lt;/td&gt;
&lt;td&gt;1.37&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Greedy rounding takes the closest value at step 1 and is stuck with the wrong menus at steps 2 and 3. The best path accepts a worse first step to get the right menus later. Plain 1-bit rounding to {-1, 1}, with the same 3 bits, scores 1.42: greedy trellis rounding loses to it, the searched path beats it. On real data the advantage is an average over long sequences, not a guarantee at every step.&lt;/p&gt;

&lt;p&gt;The search is the Viterbi algorithm: dynamic programming that keeps, for every state, the cheapest path ending there. Its cost grows linearly with length, and it runs once, offline, when the file is made.&lt;/p&gt;

&lt;p&gt;Pingala-8 uses Ungerboeck's 8-state set-partitioned trellis over symbols of 4 weights, at about 2 bits per weight. Decoding is cheap (read bits, update state, look up values), and a fused CUDA kernel does it inside the matrix-vector product. Expensive search once, cheap decode on every token: that asymmetry is why trellis coding suits inference.&lt;/p&gt;

&lt;h2&gt;
  
  
  How we measured fairly
&lt;/h2&gt;

&lt;p&gt;Quantization results are easy to flatter by accident. Our rules, which work for any comparison:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Equal file size, not equal "bits per weight".&lt;/strong&gt; Formats count overhead differently (scales, embeddings, layers kept at higher precision). File bytes decide what fits in GPU memory, so we compared against files of the same size within a few megabytes: llama.cpp's IQ2_XXS and ik_llama.cpp's IQ2_KT.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Same harness for every file.&lt;/strong&gt; WikiText-2 perplexity at context 4096 with the same tool and text windows. Five zero-shot quizzes (ARC-Easy, ARC-Challenge, HellaSwag, WinoGrande, MMLU) through one harness. Speed on one RTX 5070 Ti Laptop GPU, in one session.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Two kinds of quality.&lt;/strong&gt; Perplexity measures how well the model predicts real text (lower is better). Quizzes check whether answers survive.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Paired comparisons.&lt;/strong&gt; Every file scores the same windows and questions, so we compare item by item; shared difficulty cancels and error bars tighten.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Say what was tuned.&lt;/strong&gt; The Qwen3-4B file is the last of many builds we checked on WikiText-2, so treat it as tuned. For Llama-3.1-8B and Qwen3-14B we wrote predictions down before running.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Results
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fy44q00pkbc0ob2o9oglk.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fy44q00pkbc0ob2o9oglk.png" alt="WikiText-2 perplexity against file size for Qwen3-4B and Llama-3.1-8B" width="800" height="252"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Perplexity against file size. B4 is the released Pingala-8 file. Left: Qwen3-4B, where "post hoc" marks the tuned result. Right: Llama-3.1-8B scored without a start token in each window; the figures below add one.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Qwen3-4B, WikiText-2 perplexity at context 4096 (lower is better):&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;File&lt;/th&gt;
&lt;th&gt;Size&lt;/th&gt;
&lt;th&gt;Perplexity&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Original, fp16&lt;/td&gt;
&lt;td&gt;8.05 GB&lt;/td&gt;
&lt;td&gt;12.46&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;llama.cpp IQ2_XXS&lt;/td&gt;
&lt;td&gt;same as ours&lt;/td&gt;
&lt;td&gt;28.61&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;ik_llama.cpp IQ2_KT&lt;/td&gt;
&lt;td&gt;same as ours&lt;/td&gt;
&lt;td&gt;18.15&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Pingala-8&lt;/td&gt;
&lt;td&gt;1.24 GB&lt;/td&gt;
&lt;td&gt;16.64&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;On the five quizzes, Pingala-8 averages 50.8 against 41.9 for IQ2_XXS (+21%) and keeps 90% of the 8-bit Q8_0 average (56.3).&lt;/p&gt;

&lt;p&gt;Speed for one user (batch 1), Qwen3-4B on the RTX 5070 Ti Laptop:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Measure&lt;/th&gt;
&lt;th&gt;vs IQ2_XXS (same size)&lt;/th&gt;
&lt;th&gt;vs Q4_K_M (twice the size)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Generating text&lt;/td&gt;
&lt;td&gt;1.10x&lt;/td&gt;
&lt;td&gt;1.39x&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Reading the prompt&lt;/td&gt;
&lt;td&gt;0.74x&lt;/td&gt;
&lt;td&gt;-&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Generating one token at a time mostly means reading every weight from memory, so fewer bytes give more tokens per second while decoding stays cheap. Reading a prompt processes many tokens at once and is limited by compute, which is where the extra decode work shows.&lt;/p&gt;

&lt;p&gt;On Llama-3.1-8B (a 2.39 GB file), Pingala-8 scores 8.44 against 9.36 for IQ2_KT at the same size.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where it loses
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Qwen3-14B.&lt;/strong&gt; IQ2_KT wins, 9.40 against our 9.57. We had predicted a win in writing before the run.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Prompt speed.&lt;/strong&gt; 0.74x IQ2_XXS, so long prompts take longer before the first word. Serving several chats at once is also much slower; this is a one-user, one-GPU format.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;CUDA only.&lt;/strong&gt; It needs our patched llama.cpp and an NVIDIA GPU. Stock llama.cpp, Ollama and LM Studio cannot load the files.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Retrained models.&lt;/strong&gt; Ternary Bonsai 4B scores 11.59 at 1.07 GB, smaller and better. It is a retrained model, not a compression of the original Qwen3-4B, so it took a training run. Post-training quantization is for models nobody has retrained for you.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What the ablation says
&lt;/h2&gt;

&lt;p&gt;An ablation removes one part and measures the damage. We removed the trellis memory at equal bytes. The cost was 0.076 nats per token of extra loss, about 8% higher perplexity (e^0.076 is about 1.08).&lt;/p&gt;

&lt;p&gt;Set it against our two gaps:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Against the tuned IQ2_KT file, itself a trellis format, the trellis memory accounts for &lt;strong&gt;most&lt;/strong&gt; of our margin.&lt;/li&gt;
&lt;li&gt;Against IQ2_XXS, it accounts for only &lt;strong&gt;14%&lt;/strong&gt; of our gain. The other 86% comes from the rotation, the LDLQ rounding and the rest of the pipeline.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The lesson for anyone building a 2-bit quantizer: rotation and error-feedback rounding carry most of the quality. Trellis memory explains most of the small margin between formats that are already close, and little of the large gap to a plain codebook format. We expected the trellis to be the main story; the ablation says otherwise, which is why it is worth running.&lt;/p&gt;

&lt;h2&gt;
  
  
  How to reproduce
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Try it:&lt;/strong&gt; download the one-click build (links below), unzip, and run &lt;code&gt;run_qwen3_4b.bat&lt;/code&gt; on Windows or &lt;code&gt;./run.sh qwen&lt;/code&gt; on Linux. It downloads the model once and opens a local chat page. You need an NVIDIA RTX 30, 40 or 50 series GPU.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Rerun our numbers:&lt;/strong&gt; the repository has the patch, the commands and the logs behind every number here.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Measure your own quant the same way&lt;/strong&gt; with stock llama.cpp tools, on one machine, for files of the same size:
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# perplexity: same text file and context give the same windows&lt;/span&gt;
./llama-perplexity &lt;span class="nt"&gt;-m&lt;/span&gt; model.gguf &lt;span class="nt"&gt;-f&lt;/span&gt; wiki.test.raw &lt;span class="nt"&gt;-c&lt;/span&gt; 4096 &lt;span class="nt"&gt;-ngl&lt;/span&gt; 99

&lt;span class="c"&gt;# batch-1 speed: prompt processing (pp) and text generation (tg)&lt;/span&gt;
./llama-bench &lt;span class="nt"&gt;-m&lt;/span&gt; model.gguf &lt;span class="nt"&gt;-p&lt;/span&gt; 512 &lt;span class="nt"&gt;-n&lt;/span&gt; 128
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Why the name
&lt;/h2&gt;

&lt;p&gt;Pingala's Chandahsutra, an ancient Sanskrit work on poetic metre, lists the 8 patterns of three long or short syllables, the ganas. Our trellis has 8 states, so we named the format after him. The name is a credit; the trellis labels are Ungerboeck's.&lt;/p&gt;

&lt;h2&gt;
  
  
  Links
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Code, patch and logs: &lt;a href="https://github.com/Genrius-AI/pingala8-llm" rel="noopener noreferrer"&gt;https://github.com/Genrius-AI/pingala8-llm&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;One-click build (Windows and Linux): &lt;a href="https://github.com/Genrius-AI/pingala8-llm/releases/latest" rel="noopener noreferrer"&gt;https://github.com/Genrius-AI/pingala8-llm/releases/latest&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Paper (DOI): &lt;a href="https://doi.org/10.5281/zenodo.23481757" rel="noopener noreferrer"&gt;https://doi.org/10.5281/zenodo.23481757&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Demo video, a real-time run with the chat window recreated: &lt;a href="https://github.com/Genrius-AI/pingala8-llm/releases/download/v2.4-oneclick/Pingala8-demo-1080p.mp4" rel="noopener noreferrer"&gt;https://github.com/Genrius-AI/pingala8-llm/releases/download/v2.4-oneclick/Pingala8-demo-1080p.mp4&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Background reading: QuIP, QuIP#, QTIP, and Marcellin and Fischer's trellis-coded quantization.&lt;/p&gt;

&lt;p&gt;Written by Harshit Goyal, founder of Genrius AI (ex-Google software engineer, IIIT Delhi alumnus).&lt;/p&gt;

&lt;p&gt;&lt;em&gt;This article was written with the help of AI; every number comes from our measured logs and the paper.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>machinelearning</category>
      <category>opensource</category>
    </item>
  </channel>
</rss>
