How to run a local AI model that actually works on old hardware

I have been running language models locally on a 2012 laptop with 4 cores, 8 GB of RAM, and no dedicated graphics card. This is not a story about buying better hardware. It is a record of what actually works when you have none.

The short version: pick the right quantisation, cap the context, and stop trying to run models that were designed for machines you do not own.

The mistake everyone makes first

The obvious approach is to download a 70B model because that is what the benchmark charts show. On a machine with 8 GB of RAM, a 70B model in any useful quantisation will either fail to load or crawl at a fraction of a token per second. The process gets swapped to disk, the fans run at full speed, and the machine becomes unusable for anything else.

The correct starting point is much smaller. A 7B to 9B model at Q4 or Q3 quantisation fits in 6 to 8 GB of RAM and runs at usable speed on CPU alone. That sounds like a compromise. In practice it is the difference between a tool you use and a project you abandon.

What quantisation actually means

A model is a large matrix of numbers. Full precision stores each number in 16 or 32 bits. Quantisation stores the same weights in fewer bits, at some cost to accuracy.

The naming is not obvious. Q4_K_M means roughly four bits per weight, with a specific scheme for the outliers. Q5 is slightly larger and slightly better. Q2 is tiny and noticeably worse. The sweet spot for a memory-constrained machine is usually Q4_K_M, because the scheme spends its bits where the model needs them most.

The practical rule: if the file is larger than about 60 percent of your available RAM, it will not run well. Leave room for the operating system, the context window, and the rest of your work.

Context length is the hidden cost

People think about model size and forget about context. Every token in the context window costs memory, and that cost scales with the model. A 7B model at a 32K context can use as much memory as a much larger model at a short context.

The fix is to set the context window deliberately. Most local inference tools default to something generous, like 32K or 128K. On constrained hardware, set it to 4096 or 8192 and see whether the answers are still good. They usually are, for most tasks, and the memory saving is large.

If a task genuinely needs a long document, load it in chunks rather than widening the context. The result is slower but it completes.

What runs well on CPU only

CPU inference is not dead, it is just different. You will get somewhere between 5 and 20 tokens per second on a modern multi-core CPU, depending on the model size and quantisation. That is enough for writing, summarising, and answering questions. It is not enough for real-time chat.

Two settings matter more than anything else. Thread count should be set to your physical core count, not your logical count, because hyperthreading usually hurts inference throughput. And the batch size should stay small; large batches help throughput on GPU and do nothing useful on CPU.

The models worth trying first

Start with a small instruct model that was trained to follow instructions well. A 7B or 8B instruct model with good training will outperform a larger base model on most practical tasks, because the larger model will not load at all.

Look for models that report their quantisation clearly in the filename, and prefer those that list the context length they were trained for. A model trained on a short context and stretched to a long one performs worse than a model trained for the length you actually need.

Where to get them

Model files are hosted on Hugging Face, which is public and does not require an account for most repositories. Download the specific quantisation you want rather than the full precision file, and verify the file size matches what the page claims before you spend time on it.

What I would do differently

If I were starting again, I would spend a day benchmarking three small models at two quantisations each, and pick the one that gave me the best answers per watt rather than the biggest model that would load. The benchmarking takes an hour and saves weeks of frustration.

The other thing I would do is stop treating the local machine as a substitute for a cloud API. A local model is useful for privacy, for offline work, and for tasks where you do not want to send data anywhere. It is not a replacement for a well-hosted large model when you need the best possible answer. Knowing which situation you are in is most of the skill.