DEV Community

Tobias Reithmeier
Tobias Reithmeier

Posted on Originally published at tobiasreithmeier.de on

Local LLMs on the Mac: What Apple Silicon Realistically Delivers

#ai #llm #mac

ChatGPT, Claude and Gemini share a side effect: everything you type leaves your machine. Yet the Mac, of all computers, is one of the best machines for running language models entirely locally - no cloud, no account, no prompt ever leaving the house. What's realistic, and where does it get sluggish?

Why Apple Silicon is suited for this

Two properties make the difference. First, unified memory : CPU and GPU share the same RAM. A Mac with 32 GB can load a 28 GB model straight into GPU inference - without the back-and-forth copying between VRAM and main memory that PC setups with limited graphics memory stumble over.

Second, memory bandwidth , the real bottleneck during generation: for every token produced, the model weights have to be streamed through memory once. That's why raw GPU power matters less than GB/s here - and between a base M4 (around 120 GB/s) and an M4 Max (around 546 GB/s) lies a factor of four to five in token rate. An M4 Max with 64 GB runs a 70-billion-parameter model at a comfortably readable ~12 tokens per second.

What runs on how much RAM

The rule of thumb: a model in 4-bit quantization occupies roughly half a gigabyte per billion parameters, plus room for context and the system. That yields an honest staircase:

  • 16 GB: the 7-8B class (small Llama, Qwen, Gemma) runs briskly and is fit for everyday use. OpenAI's gpt-oss-20b is officially specced for 16 GB - but in practice that's the absolute floor: as soon as macOS and a browser compete for the memory, it becomes painfully slow
  • 32 GB: the sweet spot for ambitious beginners. The 20-30B class runs comfortably, gpt-oss-20b fluently, with headroom left for your actual work
  • 64 GB and up: the 70B class becomes usable - models that needed data center hardware two years ago now run in a MacBook. With 128 GB, large mixture-of-experts models fit too

The tools

  • Ollama is the easiest entry point: install, ollama run gpt-oss:20b, done. Model library included, officially documented by OpenAI
  • LM Studio (lmstudio.ai) offers the same with a graphical interface, model browser and local API server - good for comparing models
  • MLX, Apple's own open-source framework for Apple Silicon, often squeezes out extra speed with mlx-lm and can even fine-tune models locally

What local models are genuinely good at

Local models in the 8-30B class have become surprisingly capable at: summarizing and rephrasing, translation, questions against your own documents, code completion and commit messages, structured extraction from text. But the real trump card is something else: confidential stays confidential. Contract drafts, health questions, customer data, unreleased code - everything you wouldn't want to entrust to a cloud service is simply a non-risk with a local model. On top of that: it works offline, on a plane just as well as behind a restrictive corporate firewall, and costs nothing more after buying the Mac.

Where the limits are

Honesty is part of the deal: a local 20B model is no replacement for the frontier models in the cloud. For complex reasoning, long research tasks and demanding code, the gap is clearly noticeable. Long contexts (entire codebases, hundred-page PDFs) are additionally a game of patience, because prompt processing on Apple Silicon is comparatively slow. And on a MacBook: sustained use pulls visibly on the battery and makes the fans audible.

The realistic view: local models don't replace the cloud, they complement it - everything private, quick and recurring runs locally, the hard nuts still go to the big model.

Conclusion

"What can my Mac handle?" can now be answered precisely: 16 GB is enough to try out the 8B class, 32 GB makes local AI fit for daily use, and from 64 GB it becomes seriously capable. The barrier to entry is a single Ollama command. Once you've watched a language model answer entirely without an internet connection - with not a single word leaving your machine - you understand why local AI is more than a gimmick.

Sources

Top comments (0)