Running a local LLM on a 4GB machine without a GPU (what actually works in 2026)

The 4GB RAM, no-GPU laptop is still the most common machine in the world that a developer might want to run a local model on. The honest answer in 2026: you can run a real, useful model on it, if you accept the constraints and stop fighting them.

What fits, measured on a 4GB box:

• 1 to 2B parameter models at Q4 quantization. A 1.5B model in Q4_K_M is about 1GB of weights and leaves 2 to 3GB for the OS and context. This is the sweet spot. It writes passable short prose, does decent classification and extraction, and runs at a usable token rate on a modern CPU.
• 3B models at Q4 is the ceiling you can push. Weights around 2GB, and you will feel every swap to disk once context grows. Fine for batch jobs, unpleasant for interactive use.
• Anything 7B and up is a no at full precision. A 7B Q4 is 4GB of weights before context. You can run a 7B at Q2 to Q3 on 4GB, but the quality drop from aggressive quantization is larger than the quality gap between 7B and 2B at Q4. On extraction and summarization tasks the 2B Q4 beat the 7B Q2. Do not let the parameter count flatter you.

The setup that works:

1. llama.cpp with zero GPU layers. Do not let it offload; with no GPU it just thrashes.
2. Context window: 2048 to 4096. The KV cache is the silent RAM eater. At 4096 context a small model's cache is a few hundred MB, and that is the difference between a working setup and swap.
3. Q4_K_M as the default quant. It is the best quality per byte at this size. K-quants are worth it over plain Q4_0.
4. Pin the model file locally. Every run that starts with a gigabyte download is a run that does not happen.
5. Use a CLI or router, not a web UI. Browser tabs on a 4GB machine are the first thing to die.

What these models are actually good at, on a 4GB box: structured extraction from documents, classification, short-form rewriting, running a fixed probe corpus against an endpoint, and any task where the output format is constrained. What they are bad at: long-form generation, multi-step reasoning, and anything where the context window is the product. For those, a hosted endpoint costs pennies per run, and the local model is the fallback when the endpoint is down.

The real win is the local model as the offline tier: extraction and classification run locally for free, and the interesting questions go to the API. That split is what makes a 4GB machine feel like it has more than 4GB.

The 4GB machine is not a small data center. It is a specific tool that does specific jobs well. Pick the jobs, size the model to them, and it is genuinely useful.