Context: A model's weights — the numbers it uses to reason — are normally stored at high precision, like FP16 (16-bit floating point). Quantization compresses those numbers down to fewer bits (Q8, Q4, etc.) to shrink the file size and memory footprint, trading some numerical precision for big space savings. It's a similar idea to compressing a high-resolution photo into a smaller JPEG: the file shrinks a lot, some detail is technically lost, but for most practical purposes it's hard to tell the difference.
Ran: Before pulling anything new, ran ollama show llama3.2:1b to check what Entry 01's baseline model actually was — and it turned out to already be Q8_0, not an unspecified "default" as I'd assumed back in Entry 01. Worth a correction: Entry 01 and Entry 03's 1b numbers were Q8, not full precision. Pulled two more variants for comparison: llama3.2:1b-instruct-q4_K_M (an explicit 4-bit quantization) and llama3.2:1b-instruct-fp16 (full, uncompressed precision). Loaded each in turn, sent the same test question, and captured ollama ps / ps aux for all three quantization levels.
Result:
| Q4_K_M | Q8_0 (Entry 01 baseline) | FP16 | |
|---|---|---|---|
| Download size | 807 MB | 1.3 GB | 2.5 GB |
ollama ps loaded size |
997 MB | 1.5 GB | 2.7 GB |
| Process RSS | ~0.94 GB | ~1.24 GB | ~2.57 GB |
Memory scales cleanly with bit-width across all three: roughly 0.94 GB → 1.24 GB → 2.57 GB as precision doubles from 4-bit to 8-bit to 16-bit. That's a tidier relationship than I expected going in — quantization level is a solid, predictable lever for memory sizing.
For quality, I asked all three quantization levels the same real question: "what are good free AI tools that can create simple Google Slides using instructions?" None gave a clean answer. All three produced long lists mixing legitimate tools (Canva) with tools that have nothing to do with making slides — Q4 suggested Midjourney and DALL-E, FP16 added Artbreeder, Deep Dream Generator, and Prisma to the same mix. Full precision didn't fix this: FP16's list was just as padded with irrelevant suggestions as Q4's, and Q4 wasn't noticeably worse.
Takeaway: Memory scaled cleanly and predictably with quantization level — roughly doubling at each step from Q4 to Q8 to FP16, matching the bit-width math almost exactly. Answer quality told a different story: going all the way to full precision didn't reduce hallucinated suggestions on this open-ended recommendation question, and Q4 wasn't visibly worse than FP16. That's not proof quantization is "safe" in general — this was one soft question, not a rigorous eval — but it's a reminder that the real quality gap probably shows up on harder tasks (math, code, precise instruction-following), not casual recommendation queries. Also worth remembering going forward: check ollama show <model> before assuming what quantization a "default" pull actually gives you.
$ ollama show llama3.2:1b
quantization Q8_0
$ ollama show llama3.2:1b-instruct-q4_K_M
quantization Q4_K_M
$ ollama ps
NAME ID SIZE PROCESSOR CONTEXT UNTIL
llama3.2:1b-instruct-fp16 2887c3d03e74 2.7 GB 100% GPU 4096 4 minutes from now
$ ps aux | grep ollama
flyers 10763 0.3 15.7 438101456 2630128 ?? S llama-server --model ... -c 4096
flyers 2256 0.0 0.3 436826160 56448 ?? S ollama serve
$ ollama ps
NAME ID SIZE PROCESSOR CONTEXT UNTIL
llama3.2:1b-instruct-q4_K_M 22bc6b92eb01 997 MB 100% GPU 4096 4 minutes from now
$ ps aux | grep ollama
flyers 10786 0.2 5.7 436484688 963808 ?? S llama-server --model ... -c 4096
flyers 2256 0.0 0.4 436826160 60400 ?? S ollama serve
Top comments (0)