DEV Community

The Operator
The Operator

Posted on Edited on Originally published at coldriver8.gumroad.com

Why Docker Can’t Use Your Mac’s GPU — and How to Pool Apple Silicon Macs for LLMs Instead

#llm #macos #ansible #selfhosted

Correction (October 2026): an earlier version of this post described a Docker-based "unified GPU pool" on Apple Silicon. That doesn't work: Docker on macOS can't use the Apple GPU. The post below has been rewritten to describe the setup that actually works, which is also what the kit contains.

You have two or three Apple Silicon Macs sitting idle — a Mac Studio, a MacBook Pro, an M1 Mini. Their unified memory is great for running LLMs, so the obvious idea is to pool them. And the obvious first step most tutorials take is Docker.

On a Mac, that's a dead end for inference.

Why Docker can't use the Mac GPU

Docker Desktop, Colima and OrbStack all run your containers inside a Linux virtual machine. That VM has no access to Metal, Apple's GPU API, and there is no GPU passthrough for Apple Silicon like --gpus all on an NVIDIA Linux box. A llama.cpp or Ollama container on macOS runs on the CPU only, usually several times slower than the same model running natively.

(There's experimental work on GPU access from Linux VMs on macOS through Vulkan translation, e.g. Podman with libkrun, but it's not what Docker Desktop does and it's still slower than native Metal.)

vLLM has the same problem: its standard builds target CUDA/ROCm and don't use Metal, so "vLLM in Docker on a Mac" means CPU inference.

What works instead

Run the inference server natively on each Mac and use plain tooling around it:

Layer What runs Why
Inference llama-server (llama.cpp) built natively on each Mac Full Metal acceleration, unified memory
Supervision launchd agents Restarts on crash and reboot, nothing extra to install
Routing nginx on one "head" Mac One OpenAI-compatible endpoint, least-connections, streaming-safe
Big models llama.cpp RPC backend (rpc-server + --rpc) Splits one model's layers across Macs
Automation Ansible from any machine Repeatable, versioned config

There are two ways to use the cluster:

  • Replicas (throughput): every Mac runs its own copy of a model that fits in its memory, nginx spreads requests. Use this to serve more users.
  • Split (capacity): one model is too big for any single Mac, so llama.cpp RPC places layers on several machines. Use this to run, say, Llama 3.3 70B Q4_K_M (~43 GB) across two 32/48 GB Macs. A split model is not faster than one Mac that could hold it — network hops add latency — so split only when the model doesn't fit.

A quick taste

Check what each node has:

system_profiler SPHardwareDataType | grep -E "Chip|Memory"
Enter fullscreen mode Exit fullscreen mode

Build llama.cpp with the RPC backend (Metal is on by default on macOS):

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
cmake -B build -DGGML_RPC=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build -j
Enter fullscreen mode Exit fullscreen mode

On a worker, expose its GPU over RPC (bind to a private interface — rpc-server has no authentication):

./build/bin/rpc-server -H 10.0.9.2 -p 50052
Enter fullscreen mode Exit fullscreen mode

On the head node, load the model across the head plus the worker:

./build/bin/llama-server -m models/model.gguf -ngl 99 \
  --rpc 10.0.9.2:50052 --host 0.0.0.0 --port 8080
Enter fullscreen mode Exit fullscreen mode

Optional on a dedicated inference Mac (macOS 14+): let the GPU wire more unified memory than the default, leaving at least ~8 GB for macOS:

sudo sysctl iogpu.wired_limit_mb=57344   # example for a 64 GB Mac; resets on reboot
Enter fullscreen mode Exit fullscreen mode

Two Macs linked with a Thunderbolt cable get a Thunderbolt Bridge interface; giving it static IPs and using it for RPC traffic gives far more bandwidth than gigabit Ethernet.

The kit

Turn 2+ Macs Into One LLM Server (the Apple Silicon LLM Cluster Kit) packages all of this:

  • Why Docker can't use the Mac GPU, and the native architecture to use instead
  • Ansible inventory + playbooks: bootstrap (deps, llama.cpp build, model download), deploy, rolling model update, teardown
  • launchd templates for llama-server and rpc-server that survive reboots
  • nginx load-balancer config (least-connections, streaming-safe)
  • llama.cpp RPC setup for splitting one model across Macs
  • Memory-sizing rules, benchmark and troubleshooting tables

EUR 24.99 → https://coldriver8.gumroad.com/l/xayfxz

AI-assisted content - reviewed by author.

Top comments (0)