A single M2 Max Mac Studio can serve 8‑token‑per‑second Llama‑2‑7B inference, yet most developers leave that compute idle overnight. ## Inventory and Network Topology
Start by enumerating every Apple Silicon machine on the LAN. arp-scan --localnet | grep -i apple returns MAC addresses and hostnames. In a typical lab you might see two M2 Max units (38‑core GPU, 400 GB/s bandwidth), one M3 Ultra (80‑core GPU, 800 GB/s), and one M1 Pro (16‑core GPU). Record each host, its GPU core count, and unified memory size in a YAML inventory. Then run exo cluster init --config inventory.yaml on every node so Exo can build a peer‑to‑peer mesh over mDNS. Action: Create an inventory YAML and run exo cluster init --config inventory.yaml on each Mac to join the cluster. ## Unified Memory as a Shared Tensor Buffer
Apple’s unified memory lets the CPU and GPU address the same physical pages, eliminating PCIe copies. llama.cpp 0.3.0 exploits this with --n-gpu-layers. On an M2 Max you can offload 35 of 32 layers for a 7 B model: ./llama-server -m model.gguf -c 4096 --n-gpu-layers 35 --port 8080. The same binary on an M3 Ultra can push all 32 layers (--n-gpu-layers 32) because the 80‑core GPU holds the full weight set. Verify layer residency with llama-server -m model.gguf --verbose 2>&1 | grep "gpu_layers". Action: Set --n-gpu-layers to the maximum layers that fit in GPU memory on each model. ## Orchestration with Ray and Exo
Ray handles request routing while Exo shards the model across the mesh. Launch the head on the most powerful node: ray start --head --port=6379 --dashboard-host=0.0.0.0. On every worker run ray start --address=<head-ip>:6379 --resources='{"GPU": 1}'. Deploy a Ray Serve entry point that forwards each prompt to the actor with the lowest queue length:
@serve.deployment(ray_actor_options={"num_gpus": 0.25})
class Llama:
def __init__(self):
self.llm = LlamaCPP(model_path="model.gguf", n_gpu_layers=35)
async def __call__(self, request):
return self.llm(request.json()["prompt"])
Action: Deploy a Ray Serve deployment that routes requests to the least‑loaded GPU using ray_actor_options={'num_gpus': 0.25}. ## Monitoring, Autoscaling, and Cost Accounting
Collect GPU utilization with Apple’s powermetrics: sudo powermetrics --samplers gpu_power -i 1000 -n 1. Feed the output to a custom Prometheus exporter (mac_gpu_exporter) that scrapes every 10 seconds. Four Macs drawing ~120 W total cost roughly $0.015 /kWh at US average rates, so a 24‑hour run costs under $0.04. Add an alert rule: avg_over_time(mac_gpu_utilization[5m]) > 0.8 to trigger autoscaling via a simple launchd job that wakes sleeping Macs. Action: Add a Prometheus scrape config for mac_gpu_exporter and alert when average utilization exceeds 80% for 5 min. Start tonight by running exo cluster init on every idle Mac and point a single Ray Serve endpoint at the cluster; you will have a working distributed inference pool before the next morning.
If this was useful: I packaged a deeper version as Prompt Engineering for AI Roles: Hiring Manager & Engineer Guide.
This article was drafted with AI assistance.
Top comments (0)