4-Bit Quantization Made My Old Laptop Useful Again

I picked up a used ThinkPad T14 with only 16 GB of RAM last month. On paper it shouldn't run anything interesting — certainly not an AI model. But after learning about quantization, I've been running Llama 3.2 locally and it surprised me.

Quantization is just a fancy word for rounding numbers down to save space. A normal float16 weight takes two bytes. Quantize that to int4 and you're at half a byte per weight. That's a fourx reduction, roughly. The model gets slightly dumber in the process — but not by much.

Here's what happened when I actually tried it:

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
cmake -B build -DCMAKE_CXX_COMPILER=g++
cmake --build build --target main -j$(nproc)

Download a 4-bit quantized GGUF file (the Ollama model registry has plenty), point main at it, and you're chatting locally. No API key. No telemetry going back to some data center in Virginia.

The quality? For code review, summarizing RFCs, brainstorming architecture trade-offs — it's genuinely useful. I'd say it's about 70-80% as good as the full-precision model for most tasks. And honestly, for 90% of what I use AI for at work, that gap doesn't matter.

Where this gets interesting is the workflow angle. I used to open a browser tab, paste code into Claude or Copilot, wait for the response, copy it back. Now everything happens in my terminal. No network latency, no accidental data leakage if my connection drops mid-sentence, and zero monthly subscription. The only cost was that ThinkPad on eBay.

If you've been curious about running LLMs locally but got intimidated by the hardware requirements — start with a quantized model. You don't need an H100. You don't even need a GPU (though one helps). A recent CPU with decent RAM is enough to get started and see if local AI fits your workflow.

I put together a guide covering everything from picking quantized models to running Open WebUI as a lightweight frontend — all self-hosted, no cloud required. It's available here: https://symshah.gumroad.com/l/selfhosted-ai-homelab