TL;DR — llama.cpp is the C++ project that made running open-weight LLMs on ordinary hardware routine, via the GGUF format, block quantization, and CPU+GPU offload. This episode breaks down what those three mechanisms actually do and when reaching for llama.cpp directly beats a wrapper like Ollama.
Before there was a one-command front door, before there were hosted quant repos on every model card, there was a single C++ file that could load a Meta checkpoint and run it on a MacBook's CPU. That project is llama.cpp, and it is the reason "open-weight" and "runs on your laptop" became the same sentence. Everything in this series that mentions GGUF, quantization levels, or GPU offload is standing on this codebase.
The origin problem it solved
When the first LLaMA weights leaked, they came in PyTorch checkpoints that assumed a CUDA runtime, a Python environment, and enough VRAM to hold FP16 weights. A 7B model in FP16 is roughly 14 GB — more VRAM than most consumer GPUs had at the time. llama.cpp rewrote the inference path in plain C/C++ with no dependencies, and got that same 7B model running in under 4 GB, on CPU, at conversational speed, according to a technical breakdown from moderncpp.dev. The project has since passed 105,000 GitHub stars, per the same writeup, but the star count is a symptom, not the point: it's the reason every subsequent open-weight release ships (or gets community-converted to) a GGUF file within hours.
GGUF: the file format that made this portable
GGUF is a flat binary — a header of tensor metadata followed by tensor data laid out at page-aligned offsets. No protobuf, no HDF5, no JSON-inside-a-zip (that's what PyTorch's .pt format is doing, for comparison). The metadata parses in one pass, and because the tensor data sits at aligned offsets, llama.cpp can hand the whole file to mmap() instead of reading it into memory. The kernel maps the file into the process's address space and pages are pulled in on demand — model "loading" becomes closer to a TLB lookup than a disk read, as explained in the same moderncpp.dev piece. That's a big part of why llama.cpp starts serving a multi-gigabyte model almost instantly on a warm page cache.
Quantization: where the memory actually goes
GGUF's quantization formats compress weights in blocks. Take Q4_0, the simplest case: GGML groups weights into blocks of 32, computes one FP16 scale factor per block, and stores each weight as a 4-bit integer. That's 18 bytes for 32 weights — 4.5 bits per weight, a 3.6x compression over FP16 and 7.1x over FP32, per moderncpp.dev's breakdown. A 7B model in Q4_0 lands around 3.8 GB, which is why it fits on a cheap laptop's RAM.
In practice you'll reach for the K-quants rather than the legacy formats. A guide from Acing AI lays out the sweet spot plainly: Q4_K_M is roughly a quarter of full-precision size with minimal quality loss and is the sane default; Q5_K_M or Q6_K close the remaining gap if you have spare VRAM; Q8_0 is near-lossless but doubles Q4's footprint, which usually means you should have run a bigger model at Q4 instead. As a memory rule of thumb, Internals Decoded puts Q4_K_M at roughly 0.6–0.7 GB of VRAM per billion parameters — a 7B model fits a 6 GB GPU, a 34B model wants about 20 GB.
The KV cache adds up separately from the weights, especially at long context. llama.cpp lets you quantize it too — --cache-type-k q8_0 --cache-type-v q4_0 roughly halves KV cache memory with little quality cost, per both Internals Decoded and Acing AI. That flag alone is often the difference between a model fitting and OOM-ing on a long prompt.
CPU+GPU offload: fitting models bigger than your VRAM
The flag that matters most when a model doesn't fully fit is -ngl (number of GPU layers). Set it to 99 and llama.cpp puts as many layers on the GPU as fit, running the rest on CPU at a speed penalty — check the startup log for how many layers actually landed on GPU, as Acing AI notes. For mixture-of-experts models this gets more surgical: a Hugging Face guide on MoE offload describes assigning the "always active" parameters — attention, dense FFN, shared expert FFN — to the GPU, while routing the sparse routed-expert FFN weights to CPU via a regex like -ot "blk\.([0-9]|[1-2][0-9]|30)\.=CUDA0,exps=CPU". Because the GPU is far faster at prompt processing, llama.cpp will opportunistically copy CPU-resident weights to GPU for large batches — controlled by -b (logical batch size) and -ub (physical batch size), both often pushed to 4096 for CPU+GPU MoE setups per that same guide.
When to reach for llama.cpp directly
Ollama wraps llama.cpp for a reason — most people don't need to hand-tune tensor offload regexes. But there's a real set of jobs where the wrapper gets in the way:
Exact quant control. You want Q5_K_M for one model and Q4_K_M for another, or you want to test whether the extra bits actually change your output quality.
llama-server -hf model:Q6_Kpulls and serves a specific quant directly from Hugging Face, per the project's own docs.Odd or constrained hardware. llama.cpp treats Apple Silicon as a first-class citizen via Metal, supports AVX/AVX2/AVX512/AMX on x86, and even RVV/ZVFH support for RISC-V — per the project README. If your target is an edge box, an old workstation, or a non-NVIDIA GPU via Vulkan or SYCL, llama.cpp's backend list is the actual compatibility answer.
Squeezing a model that's slightly too big. Partial GPU offload with
-ngl 30and the rest on CPU, or MoE expert splitting with-ot, lets you run a 27B or 35B-A3B model on a 16–24 GB card that a naive full-GPU load would reject outright.No Python, no dependency chain. A plain C/C++ binary is easier to vendor into an embedded pipeline, a container with a minimal base image, or a CI job than a PyTorch stack.
Long-context serving under memory pressure. Setting
-cexplicitly instead of trusting the advertised max context, combined with--cache-type-k q8_0 --cache-type-v q8_0, is how you serve 32K context on hardware that would otherwise OOM at the model's full window — a pattern Acing AI walks through directly.
Where llama.cpp is the wrong tool: high-throughput multi-user serving at scale, where a batching-optimized engine like vLLM will out-throughput it on the same GPU; and quick one-off experimentation, where Ollama's single command is genuinely faster to get to a working model. llama.cpp earns its complexity when you need control over exactly where every byte of memory goes.
Credits & sources
The core project is ggml-org/llama.cpp on GitHub, built on the ggml library, maintained by its open-source contributor community. The internals breakdown on GGUF layout, mmap loading, and Q4_0 quantization math comes from Modern C++ // dev. The VRAM-per-billion-parameters rule of thumb and KV cache quantization notes are from Internals Decoded's piece on Ollama and llama.cpp internals. The quant selection guidance, sizing table, and long-context flag walkthrough are from Acing AI's local-serving setup guide. The CPU+GPU MoE expert-offload technique and launch command are from a Hugging Face blog post by Doctor-Shotgun.
Tomorrow's episode looks at NVIDIA's Nemotron 3 Nano 30B, a mixture-of-experts model with only about 3B active parameters at a time — another entry in this series' running theme of getting big-model quality out of small compute budgets.
Appendix — the field in one chart
Top comments (0)