DEV Community

The Operator
The Operator

Posted on Edited on

Building a Distributed LLM Cluster from Idle Apple Silicon Macs (Corrected Guide)

#llm #macos #selfhosted #llamacpp

Correction (October 2026): an earlier version of this post contained invented commands (exo cluster init --config), an impossible llama.cpp setting ("35 of 32 layers"), a Ray GPU-scheduling setup that doesn't apply to Apple GPUs, unsupported performance and electricity figures, and a link to an unrelated product. It has been rewritten; everything below uses real tools and flags.

Most developers with more than one Apple Silicon Mac leave the extra machines idle. Because the CPU and GPU share one pool of unified memory, even a mid-range Mac can hold a quantized model that would need a large discrete GPU elsewhere. Here is how to turn several of them into one LLM service, using tools that actually exist today.

First, two ways to "cluster"

  • Replicas (throughput): each Mac runs its own copy of a model that fits in its memory, and a load balancer spreads requests. More users served, same latency per request.
  • Split (capacity): one model is too big for any single Mac, so its layers are spread across machines. This lets you run a bigger model; it does not make it faster than one Mac that could hold it, because every token crosses the network.

Pick replicas unless the model genuinely doesn't fit.

Inventory each node

On every Mac:

system_profiler SPHardwareDataType | grep -E "Chip|Memory"
Enter fullscreen mode Exit fullscreen mode

Write down chip and memory per node. Memory decides what each node can serve. A rough rule for Q4_K_M quantization is ~0.6 GB per billion parameters plus a few GB for context: an 8B model is a ~5 GB file, a 70B model ~43 GB.

Inference: llama.cpp, natively

Run the server natively, not in Docker: containers on macOS run in a Linux VM with no access to the Apple GPU.

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
cmake -B build -DGGML_RPC=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build -j
./build/bin/llama-server -m ~/models/model.gguf -ngl 99 -c 8192 --host 0.0.0.0 --port 8080
Enter fullscreen mode Exit fullscreen mode

Metal is enabled by default on macOS. -ngl 99 simply means "offload all layers to the GPU"; on Apple Silicon there's no separate VRAM to budget, the limit is how much unified memory macOS lets the GPU wire (adjustable on macOS 14+ with sudo sysctl iogpu.wired_limit_mb=<MB>, leaving several GB for the OS).

Replicas: one endpoint with nginx

Run llama-server on each Mac, then put nginx in front on one of them:

upstream llama {
    least_conn;
    server 10.0.1.10:8080;
    server 10.0.1.11:8080;
}
server {
    listen 8000;
    location / {
        proxy_pass http://llama;
        proxy_buffering off;          # keep streamed tokens flowing
        proxy_read_timeout 600s;
    }
}
Enter fullscreen mode Exit fullscreen mode

llama-server exposes an OpenAI-compatible /v1/chat/completions, so clients just point at port 8000.

Split: llama.cpp RPC

On each worker, expose its backend over RPC — bind it to a private interface, because rpc-server has no authentication:

./build/bin/rpc-server -H 10.0.9.2 -p 50052
Enter fullscreen mode Exit fullscreen mode

On the main node, load the model across itself and the workers:

./build/bin/llama-server -m ~/models/big-model.gguf -ngl 99 --rpc 10.0.9.2:50052
Enter fullscreen mode Exit fullscreen mode

Two Macs linked by a Thunderbolt cable get a Thunderbolt Bridge network interface; giving it static IPs and using it for RPC traffic gives far more bandwidth than gigabit Ethernet.

An alternative for split inference is exo, which discovers other exo nodes on the local network automatically and shards a model across them; follow its README for the current install steps, as the project moves fast.

Keep it running and observable

  • Supervise llama-server with a launchd agent (KeepAlive + RunAtLoad) so it survives crashes and reboots.
  • Keep inference Macs awake: sudo pmset -a sleep 0 disksleep 0, and enable Wake for network access if you want to wake them with a Wake-on-LAN packet.
  • llama-server has a /health endpoint for checks; sudo powermetrics --samplers gpu_power -i 1000 -n 1 shows GPU power on a node if you want to feed your own metrics exporter.

For cost, do the arithmetic for your own hardware: measured watts × hours × your electricity price. For example, a hypothetical cluster averaging 120 W for 24 hours uses 2.88 kWh.

If you want the full setup — Ansible playbooks, launchd templates, nginx config and RPC split mode — I packaged it as Turn 2+ Macs Into One LLM Server.

This article was drafted with AI assistance.

Top comments (0)