In the first two episodes I covered the four metrics that matter when you evaluate an open-weights model, and how to access those models through third-party providers.
This episode is about running the open weights model local, on your own hardware.
The main constraint is basically RAM. The weights file has to fit in the memory of the machine you are using. If the file is larger than the available RAM (or VRAM), performance simply vanishes, it just doesn't work. Quantization is the main metric that you need to watch, to make local use practical. Models are commonly released or converted at 2-bit, 4-bit, 6-bit or 8-bit precision. Lower bit depth produces a smaller file and usually significant loss in quality.
Two hardware lines dominate local inference right now.
Apple Silicon covers the MacBook Air, MacBook Pro, Mac Mini and Mac Studio. Memory ranges from 16 GB on the low end up to 512 GB on the highest configurations. The unified memory architecture is unusually fast for this kind of workload, which is why even mid-range Macs can feel responsive with well-quantized models.
NVIDIA is the other main option. Consumer GPUs typically offer between 8 GB and 24 GB of VRAM. Larger systems built on the Blackwell platform (DGX Spark and boards from other manufacturers that follow the same specification) provide more capacity. In some setups it is possible to connect multiple machines and pool their memory.
A rough practical map looks like this:
- 16–32 GB: 2-bit to 4-bit quantized models in the smaller-to-mid size range. Inference is usable for many everyday tasks, but very slow
- 64–128 GB: 70-billion-parameter models become comfortable. Quality can approach what people currently get from strong closed models.
- 256 GB and above: large models such as the bigger DeepSeek or Kimi releases become realistic.
None of this is cheap at the high end, and the very large configurations remain out of reach for most individuals. The mid-range, however, is already practical for a lot of real work.
Running the model is only part of the task. You still need a way to talk to it, manage sessions, and give it tools. That layer is called the harness, and it is the subject of the final episode.
Episode 3 is live. The full series playlist is here: https://youtube.com/playlist?list=PLKxL2crAoLBU&si=3nDQeNgCw51rFDuG
Top comments (0)