Model Quantization Explained: Shrinking LLMs Without Losing Their Mind

As we all know that when we try to load a 70B parameter model, how much time and space it takes just to load the model. To solve this issue, the concept of quantization was introduced.

This post is my own attempt to understand quantization deeply enough to explain it simply — from what it actually does, to the math behind how it preserves accuracy.

The Core Idea: Precision, Not Parameters

The most common misconception is that quantization means removing parameters from a model. Actually it isn't.
Concretely: quantization converts high-precision parameters (like 32-bit floats) into lower-precision representations (like 8-bit or 4-bit numbers). This reduces:

• Memory usage (fewer bits per value)
• Inference latency (less data to move, and on supported hardware, faster math)

Floating Point 101: Sign, Exponent, Mantissa

Before we move forward, you need to know what's actually inside a floating-point number. Every float is basically scientific notation in binary:

value = (-1)^sign × 1.mantissa × 2^exponent

• Sign — positive or negative
• Exponent — how big the number can get (this controls range)
• Mantissa (significand) — the precise digits within that range (this controls precision) More mantissa bits → values closer together can be told apart (finer resolution). More exponent bits → much larger or smaller magnitudes can be represented before overflow/underflow.

This is exactly what the "E4M3" / "E5M2" naming in FP8 formats tells you — it's the literal bit split between exponent and mantissa.

Common Quantization Formats

Format	Bits	Exponent	Mantissa	Typical Use
FP16	16	5	10	General inference, good precision, limited range
BF16	16	8	7	Same range as FP32, coarser precision — popular for training
FP8 E4M3	8	4	3	Weights & forward-pass activations (precision matters more)
FP8 E5M2	8	5	2	Gradients in mixed-precision training (range matters more)
NVFP4	4	—	—	Aggressive compression on newer hardware (e.g. Blackwell)

The E4M3 vs E5M2 split exists for a reason: forward-pass weights/activations tend to be well-behaved in magnitude, so the extra mantissa bit in E4M3 helps accuracy. Gradients during training can swing across huge magnitude ranges, so E5M2 trades precision for range to avoid overflowing to infinity or underflowing to zero.

---

Symmetric vs Asymmetric Quantization

At its core, quantization maps a continuous float range onto a small set of integers using a scale factor (and sometimes a zero-point).

Symmetric Quantization

Used when the value distribution is roughly centered around zero (common for weights). The float range is mapped symmetrically around zero — no offset needed.

scale = max(abs(W)) / (2^(b-1) - 1)

W_quant   = round(W / scale)
W_dequant = W_quant × scale

Where:

• W = original full-precision weight tensor
• b = target bit-width (e.g., 8 for INT8)
• max(abs(W)) = the largest absolute value in the tensor — this is the "MaxAbs" calibration you may have seen referenced Simple, fast, and cheap to compute — but wastes range if the distribution is skewed (e.g., all-positive activations after a ReLU).

Asymmetric Quantization

Used when the distribution is skewed and doesn't center around zero (common for post-activation values). It introduces a zero-point — an offset that shifts the mapped range to better fit the actual data.

scale      = (max(W) - min(W)) / (2^b - 1)
zero_point = round(-min(W) / scale)

W_quant   = round(W / scale) + zero_point
W_dequant = (W_quant - zero_point) × scale

Asymmetric quantization better utilizes the available integer range for skewed distributions, at the cost of a slightly more expensive computation (the extra zero-point term).

---

How Accuracy Is Preserved

To maintain the model's accuracy despite the loss of numerical precision, several techniques are used:

Quantization-Aware Training (QAT)

QAT simulates quantization during training so the model learns to be robust to the precision loss it will face later.

1. Fake quantization nodes are inserted into the forward pass. These take the real FP32 weight, round it to what it would look like in low precision, then immediately cast it back to FP32. The weight is still stored in FP32 — only the forward computation "feels" the rounding error.
2. This injected error flows into the loss function, same as any other forward-pass computation.
3. The catch: rounding has a derivative of zero almost everywhere, so gradients can't flow through it normally. QAT solves this with the Straight-Through Estimator (STE) — it treats the rounding step as if it were the identity function during backpropagation, letting gradients pass through unchanged.
4. Over many steps, the optimizer nudges the real weights toward values where quantization rounding hurts the loss the least. Only after training finishes are the weights actually cast down and stored permanently in low precision.

Post-Training Quantization (PTQ) with Calibration

PTQ works on an already-trained model by using a small calibration dataset.

1. Take a small, representative sample of real inputs (for an LLM: a few hundred text sequences from a general or domain-specific corpus).
2. Run them through the full-precision model in a normal forward pass (no backpropogation).
3. At each layer, record the actual distribution of values seen — weights are fixed, but activation ranges can only be known from real data.
4. Use those observed ranges to compute the scale factor (and zero-point, if asymmetric) for that layer. Representative calibration data matters: if it doesn't resemble real deployment traffic, the resulting scale factors will be miscalibrated — clipping outliers too aggressively, or wasting precision on values that never actually occur.

Two well-known PTQ methods worth naming:

• GPTQ — uses calibration data to compute layer-wise Hessian (second-order sensitivity) information and solves for quantized weights that minimize reconstruction error.
• AWQ — protects "salient" weight channels identified by activation magnitude, rather than treating all weights equally.

Hybrid / Mixed-Precision Quantization

Not every layer is equally sensitive to precision loss. The hybrid approach quantizes different layers to different bit-widths based on measured sensitivity:

• Less sensitive layers → compressed aggressively (e.g., INT4)
• Critical layers → kept at higher precision (e.g., FP16 or INT8) to preserve overall performance Sensitivity is often measured using Hessian trace or simple accuracy/perplexity sweeps per layer.

---

Does the Model Go Back to Its Original Form at Inference?

Not exactly, but there's often a dequantization step involved, and which path is used depends on the hardware:

Dequantize-then-compute: weights sit in memory as INT4/INT8 (saving storage and memory bandwidth), but are upcast back to FP16/BF16 right before the matrix multiply. This saves memory, but the actual math still runs at higher precision — so latency gains are limited.

Native low-precision compute: on hardware with dedicated support (NVIDIA Tensor Cores — Hopper for FP8, Blackwell for NVFP4), the multiply-accumulate operation happens directly in low precision, no upcast required. This is where both memory savings and latency/throughput gains show up, because the silicon itself is doing less work per operation.

So: the stored weights never "revert" — they remain permanently in the lower-precision format. What differs is whether the compute step temporarily upcasts for the math, or the hardware crunches the low-precision numbers directly. Which path gets used depends on the deployment hardware and inference engine (e.g., TensorRT-LLM, vLLM).

---

Wrapping Up

Quantization isn't one trick — it's a toolbox: pick a format based on range vs precision needs, pick symmetric or asymmetric based on your data's distribution, and pick QAT, PTQ, or a hybrid approach based on how much retraining budget you have. The right combination lets you deploy models at a fraction of the memory and latency cost, with accuracy loss that's often barely measurable.

This post is part of my ongoing series while I learn ML deployment concepts by writing about them. Feedback and corrections welcome!