GGUF, Explained: The File Format That Put LLMs on Your Laptop

TL;DR

• GGUF is a single binary file that packs a model's weights, tokenizer, and architecture metadata together, so a tool can open it and run with no side files.
• The name on the file, like Q4_K_M or Q8_0, encodes a quantization choice: how many bits each weight gets, which decides size, speed, and quality.
• For most consumer hardware, Q4_K_M is the right default. It cuts an 8B model to about 5 GB for a 1 to 3 percent quality cost.
• GGUF is the reason serious models run on hardware you already own, with your data staying on the device.

Contents

• Why does a model need a special file format at all?
• What is actually inside a GGUF file?
• What do the quantization names like Q4_K_M actually mean?
• How much quality do you actually lose?
• The fair criticism: isn't 4-bit just lossy compression that makes the model dumber?
• How do you convert a model to GGUF yourself?
• How to pick a quant for your hardware
• Questions developers are actually asking about GGUF
• The quiet format that changed who gets to run AI
• References
• About the author

Forty years ago, a single gigabyte of computer memory cost more than a house and filled a cabinet. The phone in your pocket carries six to twelve times that, and it can run a language model trained on a meaningful slice of the open internet. The thing that carried that model from a research cluster into your hand was not a new chip. It was a file format.

If you have ever downloaded a model to run locally, you have already met it. The file ended in .gguf, and a tool like Ollama or LM Studio opened it without asking you for a config folder, a tokenizer directory, or a Python environment full of pinned versions. That quiet competence hides a lot of engineering.

GGUF is worth understanding because the filename is a decision. Q4_K_M, Q5_K_M, Q8_0: each one is a different answer to a single question, which is how much quality you are willing to trade for a model that actually fits on your machine. Most people pick by guessing. You do not have to.

Why does a model need a special file format at all?

A trained model is, underneath everything, a very large pile of numbers. Billions of them. Formats like PyTorch's .pth and Hugging Face's .safetensors store mostly those numbers and little else. To turn that pile into something you can actually run, you need more: a config.json that describes the architecture, one or more tokenizer files, a generation config, and sometimes custom Python that defines how the model is wired. You keep all of it together, you match the versions, and you have a framework installed to read it.

Quantized models make this harder, not easier. When you compress weights from 16 bits down to 4, you cannot just store smaller numbers. You also have to store the scaling factors and offsets needed to reconstruct each block, and the generic formats were never designed to carry that bookkeeping cleanly. Standard model saving methods fall short with quantized models, which need to store low-bit weights alongside their scaling factors and zero-points. GGUF was the practical answer to that gap.

What is actually inside a GGUF file?

A GGUF file introduces itself before you ask it anything. Open it and the first thing you find is structure, not just data. The format is a binary file with three main parts: a header, a metadata section, and the tensor data. The header is small bookkeeping. The metadata is where the magic lives: the architecture, the hyperparameters, the tokenizer, and the quantization type all sit there as typed key value pairs. The tensor data is the weights themselves, packed and aligned so they can be read straight off disk.

That self-describing design is the whole point. The runtime inspects the metadata, understands the architecture, loads the tokenizer, and maps the tensors without relying on a separate config.json or tokenizer folder. The weights are laid out for memory mapping, which means the model loads fast because the operating system can page it in instead of copying gigabytes around first.

GGUF grew out of an older format. It was introduced as part of the llama.cpp and GGML ecosystem in August 2023, and it is now the dominant format for distributing quantized local LLMs on Hugging Face. If you are downloading a model to run today, you are almost certainly downloading a .gguf.

What do the quantization names like Q4_K_M actually mean?

This is where most people get lost, so let me save you the hour I lost staring at a Hugging Face file list. The naming scheme is genuinely user-hostile. Nothing about Q4_K_M tells a newcomer that the M means medium, or that the K is the good kind. Here is how to read it.

The leading number is roughly the bits per weight. Q8 is eight bits, Q4 is four. The full picture is more interesting than that, because there are three families.

The legacy family is Q4_0, Q4_1, Q5_0, Q5_1, and Q8_0. These use classic per-block linear quantization, where a block stores n-bit codes and either one scale (the symmetric "_0" variants) or a scale plus an offset (the "_1" variants), decoded with a single transform per block. They are simple and fast to decode. Their weakness shows at low bit widths, where one flat map per block cannot capture skewed weight distributions well.

The K-quants are the modern default: Q3_K, Q4_K, Q5_K, Q6_K, each with _S, _M, and _L variants. The K stands for a method the llama.cpp community built, and it is smarter about where it spends its bits. K-quants perform importance-weighted bit allocation: attention and output projection tensors, which disproportionately affect quality, keep higher precision, often 6 bits, while less critical feedforward layers are quantized more aggressively. The suffix is the mix. Q4_K_M (Medium) keeps more sensitive layers at 6-bit, while Q4_K_S (Small) pushes more weights to 4-bit to save a few hundred megabytes on an 8B model.

The newest family is the I-quants, written IQ2, IQ3, IQ4, with names like IQ4_XS. The IQ family reconstructs weights using a super-block scale and an importance matrix, aimed at preserving quality at low bitrates. They squeeze more model into less space, at the cost of being more sensitive to how the quant was produced.

Here is a detail worth internalizing, because it changes how you think about all of this. Text generation is memory-bandwidth-bound. The GPU spends most of its time reading weights from memory, not doing multiply-adds, so a bigger quantization means more bytes to read per token and slower output. A smaller file is not just cheaper to store. It is literally fewer bytes to haul for every word the model writes. That is why a Q4 model often types faster than its Q8 twin. It does not think faster, it reads less.

The hard numbers, for a current 8B model, look like this:

Quant	Size (Llama-3.1-8B)	Quality vs FP16	Reach for it when
FP16	~16 GB	baseline	You are benchmarking or fine-tuning
Q8_0	~8.5 GB	under 0.5% loss	Code, math, agents, and you have the VRAM
Q6_K	~6.6 GB	~1% loss	A near-lossless step below Q8
Q5_K_M	~5.7 GB	~1% loss	You have spare memory over Q4
Q4_K_M	~4.9 GB	1 to 3% loss	The default for most people
Q4_K_S	~4.7 GB	slightly worse	Short by a few hundred MB
Q3_K_M	~4.0 GB	noticeably worse	Memory is very tight
Q2_K	~3.2 GB	severe	Last resort only

File sizes are bartowski's Meta-Llama-3.1-8B-Instruct GGUFs on Hugging Face. Quality figures are approximate and vary by model and benchmark.

How much quality do you actually lose?

For a modern 8B model, moving from full precision to Q4_K_M costs roughly 1 to 3 percent on MMLU, which is imperceptible in everyday chat and writing. Q8_0 loses under half a percent. The damage only becomes obvious in the small quants, Q2 and Q3, where reasoning visibly falls apart. That is the short version, and for most readers it is the whole answer.

The longer version comes from people who measured it carefully rather than vibing it. A 2026 study titled "Which Quantization Should I Use?" ran a unified evaluation of llama.cpp formats on Llama-3.1-8B-Instruct and made a point worth repeating: these quantization schemes are largely community-driven, with new formats, effective bit estimates, and recommended usage patterns emerging through GitHub issues, pull requests, and informal benchmark reports. In other words, the folklore in those Hugging Face comment threads is doing real work, and it is now being checked against actual downstream tasks.

One concrete number anchors the speed side of the trade. The llama.cpp project publishes benchmark figures for Llama-3.1-8B, and on those numbers Q8_0 generates tokens about 29 percent more slowly than Q4_K_M. That is the bandwidth tax made visible. You pay it in tokens per second for the fidelity you buy.

The shape of that curve is the thing to remember. Below Q4 the quality drops off a cliff. Above Q4 the line goes nearly flat, so each extra gigabyte you spend buys you almost nothing you can feel.

The fair criticism: isn't 4-bit just lossy compression that makes the model dumber?

Yes, it is lossy. Pretending otherwise would be dishonest. The honest answer is that it depends entirely on where you point the model.

For chat, summarization, and drafting, the 1 to 3 percent loss at Q4_K_M is invisible in practice, and I have shipped on-device features built on exactly that trade. For code generation, math, and multi-step agent workflows, the calculus changes, because small errors compound across steps and a single wrong token early can derail a whole chain. Q8_0 loses under 0.5 percent versus FP16 while Q4_K_M loses 1 to 3 percent, a gap that is imperceptible in everyday use but can matter on precise numerical reasoning. So the criticism is fair, and the response is to match the quant to the job rather than to argue that 4-bit is free.

Here is the part the enthusiast threads tend to skip. Requantizing a file that was already quantized, or running an aggressive IQ2 on a small model, can produce output that is confidently wrong in ways an average benchmark score will not catch. The number looks fine. The behavior is not.

There is a second limit that has nothing to do with bits. Self-describing does not mean future-proof. A GGUF file is not universally compatible forever, because the runtime still has to support the model architecture and tensor types used in the file. A brand-new architecture can land on Hugging Face and your favorite tool simply cannot read it until the maintainers add support.

How do you convert a model to GGUF yourself?

The path is two steps: convert to a high-precision GGUF, then quantize it down. The llama.cpp repository ships both tools, and the official quantize guide documents the full set of options.

# 1) Convert a Hugging Face model to a high-precision GGUF (bf16)
python convert_hf_to_gguf.py ./Llama-3.1-8B-Instruct \
  --outtype bf16 \
  --outfile llama-3.1-8b-bf16.gguf

# 2) Quantize that file down to Q4_K_M
./llama-quantize llama-3.1-8b-bf16.gguf \
  llama-3.1-8b-Q4_K_M.gguf Q4_K_M

Once you have the file, running it from Python takes a few lines with llama-cpp-python. The n_gpu_layers argument is the partial-offload lever: push as many layers onto the GPU as your VRAM allows, and the rest run on the CPU.

from llama_cpp import Llama

llm = Llama(
    model_path="llama-3.1-8b-Q4_K_M.gguf",
    n_ctx=4096,
    n_gpu_layers=20,  # offload 20 layers to the GPU, run the rest on CPU
)

out = llm("Explain GGUF in one sentence:", max_tokens=64)
print(out["choices"][0]["text"])

If you would rather skip the local setup entirely, the GGUF-my-repo space on Hugging Face builds quantized files for you and stays in sync with llama.cpp. I leaned on that more than once before I had the toolchain set up properly. Worth mentioning, the first time I converted a model myself, the conversion script choked on a tokenizer it did not recognize and the error told me almost nothing useful, so do not be surprised if step one needs a second attempt.

How to pick a quant for your hardware

Stop choosing by reputation and choose by the memory in front of you. The decision is mostly mechanical once you see it framed as hardware.

A practical guide, from least to most headroom:

• On a laptop with no dedicated GPU, or with 6 to 8 GB of VRAM, use Q4_K_M. It is the most-downloaded quant for a reason, and it gives you the most model per gigabyte.
• With 12 to 16 GB of VRAM, step up to Q5_K_M, or Q6_K if it fits, for a small bump in fidelity.
• With 24 GB or more, and when you care about code, math, or tool calling, run Q8_0. The sub-half-percent loss is cheap insurance for work where errors compound.
• Short by only a few hundred megabytes? Try Q4_K_S or IQ4_XS before you abandon the model. IQ4_XS reaches near-identical perplexity at a smaller size than Q4_K_M, useful when you need to squeeze one more gigabyte out of a tight budget.

For the very large models, the math gets unforgiving. A 70B model at Q4_K_M lands around 42.5 GB and fits across two 24 GB consumer GPUs with room for the cache, while Q5_K_M overflows that by about 2 GB before you add any context. At that scale Q4_K_M is not a preference. It is the ceiling.

If your target is a phone, pair a small model in the 1B to 4B range with Q4_K_M and keep the context window modest. I have watched a Q5 build of an 8B model run out of memory on a mid-range phone while the Q4_K_M of the same model ran without complaint. The filename was the entire difference. If you want to go deeper on running models fully on-device, I covered the Flutter side of this in a separate piece.

Questions developers are actually asking about GGUF

What does GGUF stand for?

GGUF is the single-file binary format for storing large language models, created for the llama.cpp project as the successor to the older GGML format. The GG comes from Georgi Gerganov, the developer behind both. The acronym is most often expanded as GPT-Generated Unified Format, though you will also see GGML Universal File, and the official specification does not actually spell it out. Whatever you call it, a GGUF file packs the model weights, the tokenizer, and the architecture metadata into one file you can download and run with a compatible tool.

What is the difference between GGUF and safetensors?

safetensors is a safe, fast container for raw model weights, used widely for full-precision models and for training and fine-tuning. GGUF adds the architecture metadata and tokenizer inside the file and is built around quantization for efficient inference, including on CPU. Use safetensors in training and research pipelines, and use GGUF for running quantized models locally.

Which GGUF quantization should I use?

For most people on consumer hardware, Q4_K_M is the right default. It cuts an 8B model to about 5 GB with a 1 to 3 percent quality loss that is imperceptible in chat and writing. Move up to Q5_K_M or Q8_0 if you have spare memory and need higher fidelity for code or math, and drop to Q4_K_S or IQ4_XS only if you are short a few hundred megabytes.

Can GGUF models run on a GPU or only on the CPU?

GGUF runs on CPU, GPU, or both. The llama.cpp engine supports full GPU execution, CPU-only inference using AVX and ARM NEON instructions, and partial offloading where some layers run on the GPU and the rest on the CPU. The partial-offload path is what lets a model that does not fully fit in VRAM still run with GPU acceleration.

Is Q4_K_M good enough, or do I need Q8_0?

Q4_K_M is good enough for chat, writing, and summarization, where its 1 to 3 percent quality loss is invisible. Choose Q8_0 when you need maximum fidelity for code generation, math, or multi-step agent workflows, where small errors accumulate. Q8_0 loses under half a percent versus full precision, but the file runs about 1.7 times the size of Q4_K_M and generates tokens roughly 29 percent slower on llama.cpp's published Llama-3.1-8B benchmark.

How do I convert a model to GGUF?

Use the convert_hf_to_gguf.py script from llama.cpp to turn a Hugging Face model into a high-precision GGUF, then run llama-quantize to compress it to a level such as Q4_K_M. If you want no local setup at all, the GGUF-my-repo space on Hugging Face builds quantized GGUF files for you and stays in sync with the llama.cpp project.

The quiet format that changed who gets to run AI

For a couple of years, the working assumption was that capable models live in someone else's data center, reachable only through an API key and a credit card. That assumption is breaking, and a file format is part of why.

It is an unglamorous thing to credit. Formats do not trend. Yet formats are exactly where capability either spreads or stays locked behind a counter. PDF made documents portable across machines that agreed on nothing else. MP3 made an entire library fit in a pocket. GGUF is doing the same thing to trained intelligence: moving it onto hardware people already own, with their data staying on the device instead of crossing a network.

Look at the arc and the direction is clear. The next decade of on-device AI is being built on a file you can copy to a USB stick and run on a phone. Knowing what is inside that file, and what the name on it quietly costs you in quality and speed, has stopped being a niche skill for a few enthusiasts. It is becoming part of the job.

References

1. GGUF specification, ggml-org. https://github.com/ggml-org/ggml/blob/master/docs/gguf.md
2. llama.cpp quantize tool README, ggml-org. https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md
3. llama.cpp, "Difference in different quantization methods," Discussion #2094. https://github.com/ggml-org/llama.cpp/discussions/2094
4. "Which Quantization Should I Use? A Unified Evaluation of llama.cpp Quantization on Llama-3.1-8B-Instruct," arXiv, 2026. https://arxiv.org/html/2601.14277v1
5. DataCamp, "GGUF Format: A Complete Guide to Local LLM Inference." https://www.datacamp.com/tutorial/gguf-format-a-complete-guide
6. APXML, "LLM GGUF Guide: File Format, Structure, and How It Works." https://apxml.com/posts/gguf-explained-llm-file-format
7. SitePoint, "Quantization Explained: Q4_K_M vs AWQ vs FP16 for Local LLMs." https://www.sitepoint.com/quantization-q4km-vs-awq-fp16-local-llms/
8. The Kaitchup, "Choosing a GGUF Model: K-Quants, IQ Variants, and Legacy Formats." https://kaitchup.substack.com/p/choosing-a-gguf-model-k-quants-i
9. "Q4 vs Q5 vs Q6 vs Q8 Quantization: Real Quality Loss Numbers for Local LLMs." https://runaihome.com/blog/quantization-q4-q5-q6-q8-quality-loss-2026/
10. Michael Brenndoerfer, "GGUF Format: Efficient Storage and Inference for Quantized LLMs." https://mbrenndoerfer.com/writing/gguf-format-quantized-llm-storage-inference
11. Hugging Face, "GGUF" documentation. https://huggingface.co/docs/hub/en/gguf
12. llama.cpp repository, ggml-org. https://github.com/ggml-org/llama.cpp

About the author

Sayed Ali Alkamel is a Google Developer Expert in Dart and Flutter, Founder of Flutter MENA, and Manager of Digital Application Platforms at Oman Housing Bank. He has spoken at tech events across 22+ countries and shipped apps with 2.5M+ downloads. He writes about Flutter, AI, and the developer experience at dev.to/sayed_ali_alkamel.