LLM Model Quantization Explained

Large language models are memory-bound. A 70 billion parameter model in FP32 requires roughly 280 GB of GPU memory just for weights, before accounting for activations, gradients, or the KV cache. Quantization compresses these weights into lower-precision representations, making it feasible to serve advanced models on commodity hardware. For developers, understanding quantization is no longer optional. It directly shapes latency, throughput, and the hardware bill behind every API call.

What Is Quantization?

At its core, quantization is a compression technique that maps high-precision floating-point numbers to lower-precision integers or floats. The most common jump