Shrink Your LLM by 75% and (Mostly) Keep Its Brain: Quantization Explained

Pollab Das — Thu, 09 Jul 2026 07:02:01 +0000

If you've ever tried to run a large language model on your own hardware, you've probably hit the same wall: the model is huge, your GPU's VRAM is not, and suddenly a 7B parameter model that "should" fit doesn't. This is where quantization comes in — and it's one of the most impactful techniques for making LLMs actually usable outside of a data center.

This post breaks down what quantization actually does, the major approaches you'll run into, and how to think about the trade-offs when picking one for your project.

The Core Idea

Every parameter in a neural network is a number, and by default those numbers are usually stored as 32-bit or 16-bit floating point values (FP32 or FP16/BF16). Quantization is the process of representing those same numbers with fewer bits — commonly 8-bit, 4-bit, or even lower.

Fewer bits per parameter means:

Smaller model size on disk and in memory — a 4-bit quantized model can be roughly a quarter the size of its FP16 counterpart.
Lower VRAM/RAM requirements — this is often the difference between "fits on my laptop" and "needs an A100."
Faster inference in many cases, since moving less data around reduces memory bandwidth bottlenecks (which is usually the real bottleneck, not raw compute).

The catch: reducing precision means reducing the information each number carries, which introduces error. The whole game in quantization research is minimizing that error while maximizing the compression.

Why Naive Quantization Breaks Things

You might think: "just round every weight to the nearest 4-bit value and call it a day." In practice, this tanks model quality fast, for a few reasons:

Outlier weights matter disproportionately. LLMs tend to have a small number of weights with unusually large magnitudes that carry a lot of signal. Naive rounding crushes their precision along with everything else.
Error compounds across layers. A model has dozens of layers stacked on top of each other; small errors early on can amplify as they propagate forward.
Not all parts of the model are equally sensitive. Attention layers, for instance, often tolerate quantization differently than MLP layers.

This is why modern quantization methods aren't just "round the numbers" — they're calibrated, often using a small dataset to figure out how to quantize with minimal damage.

The Major Approaches

GPTQ

GPTQ (Generative Pre-trained Transformer Quantization) is a post-training quantization method that processes weights layer-by-layer, using second-order information (an approximation of the Hessian) to figure out how to round each weight while compensating for the error introduced by previous roundings. It's calibrated against a small dataset and is one of the more established 4-bit quantization methods, especially popular for GPU inference.

AWQ (Activation-aware Weight Quantization)

AWQ takes a different angle: instead of trying to quantize every weight equally well, it identifies which weights are most important based on the activations that flow through them during inference, and protects those specifically (often by scaling them before quantization). The insight is that a small percentage of weights matter disproportionately for output quality, and preserving those gives you most of the quality of full precision at a fraction of the size.

GGUF / llama.cpp-style quantization

GGUF is a file format (successor to GGML) commonly used with llama.cpp and tools built on it, like Ollama and LM Studio. It supports a range of quantization levels denoted with names like Q4_K_M, Q5_K_S, Q8_0, etc. — the letter/number combo encodes the bit-width and the specific quantization scheme used for that block. This is the go-to format if you're doing CPU or hybrid CPU/GPU inference, or running models locally through tools like LM Studio.

A rough mental model for GGUF naming:

Q2–Q3: very small, noticeably worse quality, mainly for extreme memory constraints
Q4_K_M: the most common "sweet spot" — solid quality-to-size ratio
Q5–Q6: closer to original quality, bigger files
Q8_0: barely distinguishable from FP16 in most cases, but with much less compression benefit

bitsandbytes (LLM.int8() / NF4)

bitsandbytes is a library that plugs directly into Hugging Face's transformers, offering on-the-fly quantization (both 8-bit via LLM.int8() and 4-bit via NF4 — a data type designed for normally-distributed weights, which is what most neural network weights look like). It's less about producing a standalone quantized file and more about loading a full-precision model with quantization applied at load time, which makes it convenient for training and fine-tuning workflows (it's the backbone of QLoRA, for instance).

Quick Comparison

Method	Best For	Format	Notes
GPTQ	GPU inference	Pre-quantized checkpoint	Mature, widely supported
AWQ	GPU inference	Pre-quantized checkpoint	Often better quality at same bit-width than GPTQ
GGUF	CPU/local/hybrid inference	`.gguf` file	Powers llama.cpp, Ollama, LM Studio
bitsandbytes	Fine-tuning, quick experiments	Runtime quantization	Backbone of QLoRA

Practical Guidance

A few rules of thumb that tend to hold up:

4-bit is usually the sweet spot. Below 4-bit, quality degradation becomes noticeable for most models; above 4-bit, you're paying diminishing returns in size for marginal quality gains. Q4_K_M in GGUF or standard 4-bit GPTQ/AWQ are safe defaults.
Bigger base models tolerate quantization better. A 4-bit quantized 70B model often outperforms a full-precision 7B model on the same task, because the extra parameters give the model more redundancy to absorb quantization error.
Match the format to your runtime. If you're serving via transformers/vLLM on a GPU, GPTQ or AWQ checkpoints are usually the smoother path. If you're running locally via llama.cpp, Ollama, or LM Studio, GGUF is the native format.
Always benchmark on your actual task. Perplexity numbers are a decent proxy but don't always predict how a model behaves on your specific prompts, especially for things like instruction-following or structured output.

A Minimal Example

Here's what loading a 4-bit quantized model looks like with bitsandbytes through transformers:

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch

quant_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

model = AutoModelForCausalLM.from_pretrained(
    "your-model-id",
    quantization_config=quant_config,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("your-model-id")

And if you're running locally through something like LM Studio, you typically just download a GGUF variant of the model (e.g., Q4_K_M) and point your app at the local inference server — no quantization code required on your end, since it's baked into the file you downloaded.

Wrapping Up

Quantization isn't a single technique so much as a family of trade-offs between size, speed, and quality. The good news is that for most practical applications — local assistants, RAG pipelines, prototyping — 4-bit quantization has gotten good enough that the quality hit is barely noticeable, while the accessibility gain is enormous. If you're building something that needs to run outside a well-funded GPU cluster, it's less a question of whether to quantize and more a question of which method fits your deployment target.

DEV Community: Pollab Das