DEV Community

Cover image for Running a Fully-Local AI Agent on a Mac Studio, OpenClaw + Ollama + MLX
Bruno Mello
Bruno Mello

Posted on Edited on

Running a Fully-Local AI Agent on a Mac Studio, OpenClaw + Ollama + MLX

#ai #macos #llm #localai

A real-world, copy-paste guide to running a personal WhatsApp AI agent entirely on-device on Apple Silicon, with zero per-token API billing. Two agents from one config (a full-access private assistant and a sandboxed public one), swappable local LLM backends (Ollama and MLX), local voice (TTS + STT), and LaunchAgents so everything survives reboots.

Tested on a Mac Studio M3 Ultra (96 GB unified memory), OpenClaw 2026.7.1-2, Ollama 0.24.0, mlx-lm 0.31.3, mlx 0.32.2, macOS 27.0 beta.

Updated August 2026. This agent has now been running continuously for three months, and the things that broke were not the things I expected. New material is marked [2026]. If you only read one new section, make it 6b: the kernel panic, because it will reboot your Mac and it is not your fault.

TL;DR findings

  • You don't need the cloud. A 26B-class model (Gemma 4 26B-A4B, a 4B-active MoE) is plenty for a chatty personal agent and runs comfortably in well under half of 96 GB.
  • Ollama and MLX can coexist as two OpenClaw providers; flip the agent's primary model with a one-line config change.
  • Benchmark one model at a time. Two large models resident at once throttle each other on memory bandwidth, which nearly halved throughput and produced a totally wrong "Ollama is faster" conclusion until I unloaded the idle one. In isolation the MLX build hit ~73 tok/s vs a contended ~35. [2026] I made this exact mistake again 15 months later. See §7.
  • [2026] mlx_lm.server can kernel panic your whole Mac. An open Apple GPU driver bug, triggered by the set_wired_limit call mlx-lm makes at startup. One line of Python works around it. §6b.
  • [2026] For a chat agent, tok/s is the wrong metric. Turning reasoning on raised throughput and made replies 8.8x slower to arrive. §7b.
  • MLX has no separate "warm-up" problem. mlx_lm.server loads the model at process start and holds it; the LaunchAgent keeps it alive. Ollama lazily unloads, so it needs OLLAMA_KEEP_ALIVE + a tiny warm-up ping.

0. Prerequisites 

# OpenClaw (the agent gateway)
npm install -g openclaw@latest

# Ollama (llama.cpp backend)
brew install ollama

# MLX (Apple-silicon-native inference) in an isolated venv
python3 -m venv ~/mlx-env
~/mlx-env/bin/pip install -U mlx-lm

# ffmpeg (audio transcode for STT)
brew install ffmpeg
Enter fullscreen mode Exit fullscreen mode

Throughout, replace these placeholders with your own values:

Placeholder Meaning
<YOUR_NUMBER_E164> your WhatsApp number, e.g. +15551234567
<YOUR_GATEWAY_TOKEN> a random secret (openssl rand -hex 24)
<YOUR_GROUP_ID>@g.us a WhatsApp group id (optional)
you@example.com your provider OAuth email (optional cloud fallback)
/Users/you your home directory

1. Architecture 

                 ┌───────────────────────────────────────────┐
   WhatsApp ───► │  OpenClaw gateway (loopback :18789)       │
                 │                                           │
                 │  agent "private"  ── full tools           │
                 │  agent "public"   ── sandboxed (no bash)  │
                 └───────┬─────────────────────┬─────────────┘
                         │ model providers       │
            ┌────────────▼─────────┐   ┌─────────▼──────────────┐
            │ ollama  :11434       │   │ mlx  :8080             │
            │ (llama.cpp / Metal)  │   │ mlx_lm.server (OpenAI- │
            │                      │   │  compatible endpoint)  │
            └──────────────────────┘   └────────────────────────┘
            voice:  OmniVoice TTS :17494   ·   mlx-whisper STT :17495
Enter fullscreen mode Exit fullscreen mode

One config defines two agents: a full-access private agent bound to your DM, and a locked-down public agent for everyone else.

2. The two local LLM providers

Both providers live under models.providers in ~/.openclaw/openclaw.json. Ollama speaks its native API; MLX is exposed as an OpenAI-compatible server, so OpenClaw talks to it with api: openai-completions.

{
  "models": {
    "providers": {
      "ollama": {
        "api": "ollama",
        "apiKey": "ollama-local",
        "baseUrl": "http://127.0.0.1:11434",
        "models": [
          { "id": "gemma4:26b-a4b-it-q8_0", "name": "Gemma 4 26B (Q8_0)",
            "contextWindow": 131072, "input": ["text","image"], "reasoning": true,
            "cost": { "input": 0, "output": 0, "cacheRead": 0, "cacheWrite": 0 } }
        ]
      },
      "mlx": {
        "api": "openai-completions",
        "apiKey": "mlx",
        "baseUrl": "http://127.0.0.1:8080/v1",
        "models": [
          { "id": "mlx-community/gemma-4-26B-A4B-it-qat-4bit",
            "api": "openai-completions", "name": "Gemma 4 26B-A4B QAT-4bit (MLX)",
            "contextWindow": 131072, "input": ["text"], "reasoning": true, "maxTokens": 4096,
            "cost": { "input": 0, "output": 0, "cacheRead": 0, "cacheWrite": 0 } }
        ]
      }
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

cost: 0 everywhere, these are local, free. It also keeps OpenClaw's usage accounting honest.

3. Wiring models into the agent + aliases

Under agents.defaults, register the models (with short aliases for quick swaps) and pick the primary:

{
  "agents": {
    "defaults": {
      "workspace": "/Users/you/.openclaw/workspace",
      "model": {
        "primary": "mlx/mlx-community/gemma-4-26B-A4B-it-qat-4bit"
      },
      "models": {
        "ollama/gemma4:26b-a4b-it-q8_0":          { "alias": "gemma4-26b-q8",   "params": { "think": true } },
        "mlx/mlx-community/gemma-4-26B-A4B-it-qat-4bit": { "alias": "gemma4-26b-qat", "params": { "think": true } }
      }
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

Swap Mika's brain with a one-liner (then restart the gateway):

openclaw config set agents.defaults.model.primary 'ollama/gemma4:26b-a4b-it-q8_0'
openclaw gateway restart
openclaw config get agents.defaults.model.primary   # verify
Enter fullscreen mode Exit fullscreen mode

4. Two agents from one config (private vs public)

This is the underrated trick: define a locked-down public persona alongside the full-access private one. The public agent denies the dangerous tools.

{
  "agents": {
    "list": [
      {
        "id": "private",
        "name": "Mika",
        "workspace": "/Users/you/.openclaw/workspace"
      },
      {
        "id": "public",
        "name": "Mika (Public)",
        "workspace": "/Users/you/.openclaw/workspace/public",
        "tools": { "deny": ["bash", "process", "web_search"], "exec": {} }
      }
    ]
  },
  "bindings": [
    { "agentId": "private", "match": { "channel": "whatsapp", "peer": { "id": "<YOUR_NUMBER_E164>", "kind": "dm" } } },
    { "agentId": "public",  "match": { "channel": "whatsapp" } }
  ]
}
Enter fullscreen mode Exit fullscreen mode

Your DM hits private (can run shell, browse, etc.); everyone else hits public (chat + safe tools only).

5. Ollama: tuning + keeping the model warm

[2026] Ollama's job in this stack changed. It no longer serves a chat model here: MLX won that role outright (§7), and I retired the duplicate 33 GB Q8_0 build along with the warm-up LaunchAgent below. Ollama now runs for exactly one thing, embeddings for memory search (§8c), which needs no warm-up because a 274 MB embedder loads on demand in well under a second. The section below still stands if you want a chat model on Ollama, or as a second opinion when a quant looks suspicious. Just do not keep both resident, for the reason in §7.

Ollama lazily unloads models after OLLAMA_KEEP_ALIVE (default 5 min), so the next request pays a cold-start. Two fixes: tune the service env, and pre-warm on boot.

5a. Service env (LaunchAgent: ~/Library/LaunchAgents/homebrew.mxcl.ollama.plist) 

<key>EnvironmentVariables</key>
<dict>
    <key>OLLAMA_FLASH_ATTENTION</key>  <string>1</string>
    <key>OLLAMA_KEEP_ALIVE</key>       <string>24h</string>
    <key>OLLAMA_KV_CACHE_TYPE</key>    <string>q8_0</string>
</dict>
Enter fullscreen mode Exit fullscreen mode 
launchctl unload ~/Library/LaunchAgents/homebrew.mxcl.ollama.plist
launchctl load   ~/Library/LaunchAgents/homebrew.mxcl.ollama.plist
Enter fullscreen mode Exit fullscreen mode

5b. Warm-up script: ~/ollama-warmup.sh 

#!/bin/bash
# Pre-warm an Ollama model into GPU after service start.
MODEL="${1:-gemma4:26b-a4b-it-q8_0}"
MAX_RETRIES=30; RETRY_INTERVAL=2
echo "[warmup] waiting for Ollama..."
for i in $(seq 1 $MAX_RETRIES); do
  if curl -s http://localhost:11434/api/tags >/dev/null 2>&1; then
    echo "[warmup] loading $MODEL..."
    curl -s http://localhost:11434/api/generate \
      -d "{\"model\":\"$MODEL\",\"prompt\":\"hi\",\"stream\":false,\"keep_alive\":\"24h\"}" >/dev/null 2>&1
    echo "[warmup] $MODEL warm."; exit 0
  fi
  sleep $RETRY_INTERVAL
done
echo "[warmup] Ollama did not start in time"; exit 1
Enter fullscreen mode Exit fullscreen mode

5c. Warm-up LaunchAgent: ~/Library/LaunchAgents/com.ollama.warmup.plist 

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0"><dict>
  <key>Label</key><string>com.ollama.warmup</string>
  <key>ProgramArguments</key>
  <array>
    <string>/Users/you/ollama-warmup.sh</string>
    <string>gemma4:26b-a4b-it-q8_0</string>
  </array>
  <key>RunAtLoad</key><true/>
  <key>StandardOutPath</key><string>/tmp/ollama-warmup.log</string>
  <key>StandardErrorPath</key><string>/tmp/ollama-warmup.log</string>
</dict></plist>
Enter fullscreen mode Exit fullscreen mode 
chmod +x ~/ollama-warmup.sh
launchctl load ~/Library/LaunchAgents/com.ollama.warmup.plist
# disable later (keeps the file):  launchctl unload -w ~/Library/LaunchAgents/com.ollama.warmup.plist
Enter fullscreen mode Exit fullscreen mode

6. MLX: a persistent OpenAI-compatible server

mlx_lm.server loads the model at startup and holds it for the life of the process, so the LaunchAgent is the warm-up. Pull a model once (it caches under ~/.cache/huggingface):

~/mlx-env/bin/hf download mlx-community/gemma-4-26B-A4B-it-qat-4bit
Enter fullscreen mode Exit fullscreen mode

LaunchAgent: ~/Library/LaunchAgents/com.mlx-lm.server.plist 

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0"><dict>
  <key>Label</key><string>com.mlx-lm.server</string>
  <key>ProgramArguments</key>
  <array>
    <string>/Users/you/mlx-env/bin/python3</string>
    <string>/Users/you/mlx-env/mlx_server_shim.py</string>
    <string>--model</string><string>mlx-community/gemma-4-26B-A4B-it-qat-4bit</string>
    <string>--port</string><string>8080</string>
    <string>--prompt-cache-size</string><string>8</string>
    <string>--prompt-cache-bytes</string><string>12000000000</string>
  </array>
  <key>EnvironmentVariables</key>
  <dict>
    <key>PATH</key><string>/Users/you/mlx-env/bin:/opt/homebrew/bin:/usr/bin:/bin</string>
    <key>MLX_LM_DISABLE_WIRED_LIMIT</key><string>1</string>
  </dict>
  <key>RunAtLoad</key><true/>
  <key>KeepAlive</key><true/>
  <key>StandardOutPath</key><string>/tmp/mlx-lm-server.log</string>
  <key>StandardErrorPath</key><string>/tmp/mlx-lm-server.log</string>
</dict></plist>
Enter fullscreen mode Exit fullscreen mode 
launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.mlx-lm.server.plist
curl -s http://127.0.0.1:8080/v1/models | python3 -m json.tool   # confirm it's serving
Enter fullscreen mode Exit fullscreen mode

[2026] Two changes in this plist. It no longer launches mlx_lm.server directly, it goes through a small shim that disables set_wired_limit (see §6b, not optional for a 24/7 agent), and it bounds the prompt cache. Also note bootstrap rather than the older launchctl load.

[2026] The launchd trap that cost me an hour. launchctl kickstart -k restarts the process but reuses launchd's cached job definition, so an edited plist is silently ignored. The service comes back, the port answers, the log looks healthy, and your change did nothing. After any plist edit, use launchctl bootout gui/$(id -u)/<label> then bootstrap, and verify against the running process rather than the file with ps -o args= -p $(pgrep -f mlx_server_shim.py). Corollary: because KeepAlive respawns a dead server transparently, health-check by comparing the PID, not by asking whether the port answers. A crash-and-respawn looks identical to healthy uptime from outside.

[2026] Why QAT, not OptiQ? The original version of this post used OptiQ-4bit, a mixed-precision community quant that keeps the MoE router at 8-bit and the experts at 4-bit. I have since switched to Google's official quantization-aware-trained qat-4bit, for two reasons. It is faster (~83 tok/s vs ~73, because it is uniformly 4-bit rather than keeping the router at 8-bit), and it removes a single-maintainer dependency in favour of a standard reproducible build. Same ~15-17 GB resident.

6b. [2026] The kernel panic that will reboot your Mac

This is the single most important thing I learned in three months of running this agent 24/7, and it is barely documented anywhere.

If you run mlx_lm.server as a long-lived service, your Mac can kernel panic and reboot. Not crash the Python process: reboot the machine, losing whatever else you had open.

panic(cpu 8 caller 0x...): "completeMemory() prepare count underflow" @IOGPUMemory.cpp:550
Kernel Extensions in backtrace:
   com.apple.iokit.IOGPUFamily(...)
Enter fullscreen mode Exit fullscreen mode

Tracked as ml-explore/mlx#3186, open since March 2026 and still unfixed as of late August, with no response from Apple or the MLX maintainers anywhere in the thread. Reproduced on M1 Ultra, M3 Ultra, M4, M4 Pro, M4 Max and M5, from 32 GB to 256 GB. Not a "wrong hardware" problem, and not an out-of-memory problem.

What actually triggers it

A commenter did the work everyone else skipped: a controlled single-variable isolation, ten fresh-server soak runs per arm (full report). The result overturns the obvious assumption:

  • Not prefill size. A replay of a crashing session with prefills up to 23k tokens ran 111 minutes clean.
  • Concurrency plus prompt-cache eviction churn. Two concurrent streams, plus unique prompts overflowing --prompt-cache-bytes, panics in 102 to 108 seconds, 3 of 3 cold boots.
  • Not OOM. Real exhaustion gives a clean userspace Metal error. The panic hits with headroom to spare.

For a WhatsApp agent this matters more than it looks. Overlapping messages plus scheduled cron jobs is concurrency, and a long conversation churning the prompt cache is the eviction pattern. This exact setup sits squarely inside the trigger profile.

The one mitigation that works

mlx_lm.server calls mx.set_wired_limit(...) at startup, claiming roughly 75% of RAM as wired. Wired pages cannot be reclaimed by macOS. That single call is the discriminating factor:

Change (one variable per arm) Result
none (stock) panic x3 @ 102 to 108s
set_wired_limit never called no panic, 10/10 runs, 4.2h
+ also no periodic clear_cache no panic, 10/10 runs, 5.0h
only skip periodic clear_cache panic @ 99.9s
sysctl iogpu.disable_wired_collector=1 + metal4 async mapping off panic @ 109.6s
RotatingKVCache / lower wired cap / MLX_MAX_OPS_PER_BUFFER die earlier as clean userspace OOM

Note what fails: the sysctl workarounds you will find on forums, and bounding the KV cache. Also worth knowing, --max-kv-size is silently ignored for architectures that define make_cache.

The cost of the fix is about 3% sequential throughput, because weights become pageable.

The shim

There is no upstream opt-out. As of mlx-lm 0.31.3, mx.set_wired_limit is still called unconditionally in seven places, including server.py. So wrap the server in a launcher, ~/mlx-env/mlx_server_shim.py:

#!/usr/bin/env python3
# Launcher that no-ops set_wired_limit before mlx_lm.server starts.
# Mitigation for ml-explore/mlx#3186.
import os, sys
import mlx.core as mx

def main():
    if os.environ.get("MLX_LM_DISABLE_WIRED_LIMIT") == "1":
        mx.set_wired_limit = lambda *a, **k: 0
        print("[shim] wired limit disabled", file=sys.stderr)
    from mlx_lm.server import main as server_main
    sys.argv = ["mlx_lm.server"] + sys.argv[1:]
    server_main()

if __name__ == "__main__":
    main()
Enter fullscreen mode Exit fullscreen mode

Patch at the mlx.core module level: mlx_lm shares that module object, so every call site is covered, including the one inside server.py that a narrower patch would miss. Confirm the [shim] line appears in your log. Do not assume it applied.

The symptom you are probably already living with

Long before it panics, this bug has a milder form that I tolerated for months without diagnosing: the further a session gets through a large context window, the more the whole Mac drags. Not just the model, everything.

Same root cause. The KV cache grows, wired allocations grow with it, and macOS cannot reclaim any of it, so the entire system starves. With the shim in place that memory is pageable instead: during a heavy session I watched free RAM dip to 44% and then recover to 86% once the task finished. That reclaim is exactly what wiring prevents.

If your Mac gets sluggish at high context, you are not "running a big model", you are on the road to a reboot.

7. Benchmarking: and the gotcha that almost fooled me

Same model family, same prompt, 200-token generation, steady-state, wall-clock:

Backend / build Decode speed Resident
MLX gemma-4-26B-A4B-it-OptiQ-4bit (isolated) ~73 tok/s ~17 GB
Ollama gemma4:26b-a4b-it-q8_0 ~60 tok/s ~33 GB
MLX while a second model is also resident ~30-36 tok/s (contended) 
# Ollama: exact decode rate from the API (excludes prompt + load)
curl -s http://localhost:11434/api/generate -d '{
  "model":"gemma4:26b-a4b-it-q8_0","prompt":"Count from 1 to 100 slowly.",
  "stream":false,"think":false,"options":{"num_predict":200,"temperature":0}}' \
| python3 -c 'import json,sys;d=json.load(sys.stdin);print(round(d["eval_count"]/(d["eval_duration"]/1e9),1),"tok/s")'

# MLX: wall-clock over completion_tokens (short prompt ⇒ prompt time negligible)
time curl -s http://127.0.0.1:8080/v1/completions -d '{
  "model":"mlx-community/gemma-4-26B-A4B-it-qat-4bit",
  "prompt":"Count from 1 to 100 slowly.","max_tokens":200,"temperature":0}' >/dev/null
Enter fullscreen mode Exit fullscreen mode

⚠️ The lesson: the first time I ran this, MLX clocked ~35 tok/s and I "concluded" Ollama was 1.7× faster. Wrong. The 33 GB Ollama model was still resident and the two were fighting over memory bandwidth. Unload everything but the model under test (ollama stop <model>), then measure. In its real deployed condition (only the MLX model resident) it runs faster and lighter.
[2026] I did it again, 15 months later. Upgrading MLX, I measured a 2.7x speedup: 30 tok/s before, 84 after. Wonderful, and entirely fake. A second model was resident during the "before" run and not the "after" run. With the GPU actually quiet, both versions do ~83 tok/s and the upgrade changed nothing. Knowing the trap is not the same as remembering to check for it, so build the check into the harness rather than into your intentions.

The tell is variance. A contaminated sweep gave 20.5, 36.6 and 72.4 tok/s for the same config. Wild run-to-run spread means contention, not noise. Clean runs are boring and tight, and if your numbers are not boring, stop and find out what else is resident.

7b. [2026] For a chat agent, tok/s is the wrong metric

I turned reasoning on for the agent because the benchmark said it was faster. It was, and the agent got dramatically worse to use.

With think: true and enable_thinking in the chat template, decode throughput went up: the model produces long reasoning traces at full speed, and a longer generation amortises fixed per-request overhead, so tok/s looks great. Measured on a real inbound WhatsApp message:

Reasoning Time to reply Tokens generated
off 0.73s 32
on 6.42s 529

Same question, same quality of answer, 8.8x longer to arrive. The model spent 497 extra tokens thinking about a message that needed one sentence.

Throughput measures how fast tokens come out. What a conversational agent actually lives or dies by is time to first useful reply, and reasoning trades the second for the first. For an agentic coding tool that ratio is often worth it. For someone texting you on WhatsApp, it is not.

If you want a middle ground, OpenClaw's thinkingDefault accepts off, minimal, low, medium, high, xhigh, adaptive, max. adaptive is the interesting one: reason only when the message looks like it needs it.

A related trap from the original §9: with reasoning on and a small max_tokens, the model can spend the entire budget thinking and return empty content. If you enable reasoning, raise the ceiling at the same time.

8. Bonus: fully-local voice (TTS + STT)

OpenClaw treats both as OpenAI-compatible/CLI endpoints, so no cloud and no keys.

TTS: OmniVoice on :17494 (messages.tts) 

{
  "messages": {
    "tts": {
      "provider": "openai",
      "auto": "always",
      "providers": {
        "openai": {
          "enabled": true,
          "baseUrl": "http://127.0.0.1:17494/v1",
          "apiKey": "omnivoice-local",
          "model": "omnivoice",
          "voice": "female-young-pt"
        }
      }
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

(OmniVoice runs behind a small Python wrapper that exposes /v1/audio/speech, kept alive by its own LaunchAgent, same pattern as the STT server below.)

STT: mlx-whisper on :17495

~/whisper-server.py (FastAPI wrapper around mlx_whisper, loads whisper-large-v3-turbo once):

#!/usr/bin/env python3
"""HTTP wrapper for mlx-whisper STT. Loads model once, serves many requests."""
import os, tempfile, time
from contextlib import asynccontextmanager
import mlx_whisper
from fastapi import FastAPI, File, Form, UploadFile
from fastapi.responses import JSONResponse

PORT = int(os.environ.get("WHISPER_PORT", "17495"))
MODEL_REPO = os.environ.get("WHISPER_MODEL", "mlx-community/whisper-large-v3-turbo")

@asynccontextmanager
async def lifespan(_app: FastAPI):
    # warm the model on a moment of silence so the first real request is fast
    silent = os.path.join(tempfile.gettempdir(), "whisper-warmup.wav")
    if not os.path.exists(silent):
        import wave
        with wave.open(silent, "wb") as w:
            w.setnchannels(1); w.setsampwidth(2); w.setframerate(16000)
            w.writeframes(b"\x00\x00" * 16000)
    mlx_whisper.transcribe(silent, path_or_hf_repo=MODEL_REPO)
    yield

app = FastAPI(lifespan=lifespan)

@app.get("/health")
async def health(): return {"status": "ok", "model": MODEL_REPO, "port": PORT}

@app.post("/transcribe")
async def transcribe(file: UploadFile = File(...), language: str | None = Form(None), model: str | None = Form(None)):
    started = time.time()
    suffix = os.path.splitext(file.filename or "")[1] or ".wav"
    with tempfile.NamedTemporaryFile(suffix=suffix, delete=False) as tmp:
        tmp.write(await file.read()); path = tmp.name
    try:
        kw = {"path_or_hf_repo": model or MODEL_REPO}
        if language and language.lower() not in ("auto", ""): kw["language"] = language
        r = mlx_whisper.transcribe(path, **kw)
        return JSONResponse({"text": (r.get("text") or "").strip(),
                             "language": r.get("language"),
                             "duration": round(time.time() - started, 3)})
    finally:
        try: os.unlink(path)
        except OSError: pass

if __name__ == "__main__":
    import uvicorn
    uvicorn.run(app, host="127.0.0.1", port=PORT, log_level="info")
Enter fullscreen mode Exit fullscreen mode

Install + LaunchAgent (~/Library/LaunchAgents/com.mlx-whisper.server.plist):

~/mlx-env/bin/pip install mlx-whisper fastapi 'uvicorn[standard]' python-multipart
Enter fullscreen mode Exit fullscreen mode 
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0"><dict>
  <key>Label</key><string>com.mlx-whisper.server</string>
  <key>ProgramArguments</key>
  <array><string>/Users/you/mlx-env/bin/python3</string><string>/Users/you/whisper-server.py</string></array>
  <key>EnvironmentVariables</key>
  <dict>
    <key>PATH</key><string>/opt/homebrew/bin:/usr/bin:/bin</string>   <!-- mlx-whisper shells out to ffmpeg -->
    <key>WHISPER_PORT</key><string>17495</string>
    <key>WHISPER_MODEL</key><string>mlx-community/whisper-large-v3-turbo</string>
  </dict>
  <key>RunAtLoad</key><true/><key>KeepAlive</key><true/>
  <key>StandardOutPath</key><string>/tmp/mlx-whisper-server.log</string>
  <key>StandardErrorPath</key><string>/tmp/mlx-whisper-server.log</string>
</dict></plist>
Enter fullscreen mode Exit fullscreen mode

Hook it into OpenClaw as the audio model (tools.media), transcode to WAV, POST, return text:

{
  "tools": {
    "media": {
      "audio": { "enabled": true },
      "models": [{
        "type": "cli", "command": "bash", "provider": "mlx-whisper", "model": "whisper-large-v3-turbo",
        "capabilities": ["audio"],
        "args": ["-c",
          "TMP=$(mktemp -t stt) && /opt/homebrew/bin/ffmpeg -y -i \"$1\" -f wav \"$TMP\" 2>/dev/null && curl -s -X POST http://localhost:17495/transcribe -F \"file=@$TMP;filename=stt.wav;type=audio/wav\" -F 'language=pt' | python3 -c \"import json,sys;d=json.load(sys.stdin);print(d.get('text',''))\" ; rm -f \"$TMP\"",
          "--"]
      }]
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

Tip: whisper-base hallucinates a trailing phantom phrase on short clips. whisper-large-v3-turbo (≈809M, ~1.5 GB) fixes it at ~0.3 s/clip on an M3 Ultra.

8b. [2026] Voice delivery: the duplicate-message trap

Adding TTS created a bug that took several rounds to pin down. Asking the agent for a voice note produced the audio and a text message repeating the same thing. Once, four identical audios in three seconds.

The structural cause is worth knowing before you build on this: OpenClaw implements "voice message replaces text" only for Feishu. There is a shouldSuppressFeishuTextForVoiceMedia() helper, referenced solely from the Feishu path. WhatsApp has no equivalent. If a turn produces both text and media, WhatsApp delivers both. There is no config switch for this, so it can only be fixed at the instruction level.

Three separate paths produced the same symptom, and all three had to be closed:

  1. The final turn text. The agent sent the audio, then ended the turn with a transcription. The rule has to be absolute: after sending media, end the turn with empty text. My first attempt allowed "one short, different sentence" and it used that latitude to send the transcription anyway. Leave zero escape hatches.
  2. The progress update. With path 1 closed, the order flipped: text first, audio six seconds later. It had switched to sending "one moment, recording that now", which its own instructions permitted. Progress updates need scoping to genuinely long work (multi-step research, browser automation), never to generating a file.
  3. Session history. The 4x audio loop was not fixed by instructions at all. The session store contained the looping pattern and the model was imitating its own recent behaviour. Clearing it fixed it. Note there is no openclaw sessions delete command: stop the gateway first (it rewrites stores from memory on shutdown), then write {} to ~/.openclaw/agents/<id>/sessions/sessions.json.

The general lesson: when an agent misbehaves repeatedly, check whether it is copying itself from history before you keep rewriting the prompt.

8c. [2026] Memory search needs an embedding model, and it fails hard

If you enable memorySearch, OpenClaw embeds the query before searching, so it needs a live embedding endpoint:

{ "agents": { "defaults": { "memorySearch": {
  "provider": "ollama", "model": "nomic-embed-text" } } } }
Enter fullscreen mode Exit fullscreen mode

I had this pointed at a GUI app's bundled embedder, and it was silently broken for weeks. The failure mode is what makes this worth writing down: I assumed it would degrade to keyword search, because the index is hybrid and carries both FTS and vector paths. It does not. The query path embeds first, so a dead embedder kills the entire search:

Memory search failed: fetch failed | connect ECONNREFUSED 127.0.0.1:1234
Enter fullscreen mode Exit fullscreen mode

Ollama is the right home for this. It is already in this stack, it autostarts as a LaunchAgent, and nomic-embed-text is only 274 MB:

ollama pull nomic-embed-text
openclaw memory status --deep                       # "Embeddings: ready"
openclaw memory status --index --agent private      # reindex after changing model
openclaw memory status --index --agent public
Enter fullscreen mode Exit fullscreen mode

Two things to watch. Reindex after any embedder change, since the stored index records which model built it and vector search pauses until rebuilt. And check dimensions match before switching: both nomic v1.5 builds are 768-dim, so my existing vectors stayed compatible.

Verify semantically, not with a keyword you know is present. Query something whose words do not appear in the target text and confirm you still get the right chunk back.

9. Gotchas worth knowing

  • Co-residency throttles throughput (see §7). Keep one big model resident; ollama stop <model> frees it instantly.
  • Slower model ⇒ blown cron timeouts. A research-heavy scheduled job that finished on a fast quant can time out on a slower one. Bump the job's timeoutSeconds, or give the cron its own faster model.
  • Stale "typing…" indicator. If WhatsApp's connection drops mid-turn, the "composing" presence may never get its "stop." It's cosmetic; an openclaw gateway restart clears it.
  • Reasoning models eat your token budget. With think: true and a tiny max_tokens, the model can spend the whole budget "thinking" and return empty content. Give it headroom.
  • [2026] Your Mac reboots under load. Not your fault, not your RAM. See §6b, and disable set_wired_limit.
  • [2026] Rapid gateway restarts revoke your WhatsApp session. I restarted the gateway roughly seven times in an hour while iterating on instructions. WhatsApp treated it as abuse and invalidated the linked device: [whatsapp] auto-restart skipped, terminal disconnect. The agent went fully offline until I re-scanned a QR. Batch your config edits and restart once. Worse, openclaw channels status still reports linked, because the local credentials survive and only the server side revoked them. Check running, connected and health: instead.
  • [2026] Scheduled jobs confabulate. A cron that asks for facts ("summarise today's crypto news") will happily invent them if the prompt does not name the tool it must use and demand citations. Verify against the gateway log rather than trusting the message you receive. Anything a cron reports that you cannot trace to a tool call did not happen.
  • [2026] An agent that "tested" something probably did not. Mine confidently reported benchmarks it structurally could not run. Check claimed actions against the tools the agent actually has.
  • A cloud fallback can stay configured (e.g. an OpenAI/Anthropic OAuth profile) without ever spending API credits, just don't make it primary. OAuth access tokens expire and re-auth is interactive, so don't rely on it for unattended jobs.

10. Service map (what autostarts)

LaunchAgent Purpose Port
ai.openclaw.gateway the agent gateway 18789 (loopback)
homebrew.mxcl.ollama Ollama backend 11434
com.ollama.warmup pre-warm Ollama model (optional; *[2026]** disabled here, see §5)* -
com.mlx-lm.server MLX LLM (OpenAI-compatible) 8080
com.mlx-whisper.server STT 17495
com.omnivoice.server TTS 17494
[2026] com.fm-vision.server local vision shim (image input) 17496 
launchctl list | grep -E 'openclaw|ollama|mlx|omnivoice|fm-vision'
Enter fullscreen mode Exit fullscreen mode

[2026] Verify reboot-resilience properly. launchctl list shows what is loaded now. What survives a reboot is RunAtLoad plus not being disabled, and those are different things. Check both:

plutil -extract RunAtLoad raw ~/Library/LaunchAgents/<label>.plist
launchctl print-disabled gui/$(id -u) | grep <label>

A service can be running happily today and still be => disabled, in which case it will not come back after a reboot.

Built and debugged interactively with Claude Code. Numbers are from a single M3 Ultra (96 GB); your mileage will vary with chip, RAM, and quant. Updated August 2026 after three months of continuous 24/7 operation, which is where most of the [2026] material came from: none of it showed up in the first week.

Top comments (2)

Collapse
 
harjjotsinghh profile image
Harjot Singh •

Mac Studio + MLX + Ollama is a genuinely good local-agent stack right now - unified memory means you can hold a sizable model without a discrete GPU's VRAM ceiling, and MLX is finally making Apple Silicon a real inference target instead of an afterthought. The appeal is obvious: no per-token bill, no data leaving the machine, no rate limits. For privacy-sensitive or high-volume workloads that's a real unlock.

The honest tradeoff to keep in front of readers: local buys you privacy and zero marginal cost, but you pay in capability ceiling and throughput - a local model is a step below frontier on hard reasoning, and a single Mac Studio is one concurrency lane, so it shines for personal/agentic background work and struggles for anything needing peak quality or parallelism. Which is exactly why I landed on routing instead of all-local for Moonshift, the thing I build - a multi-agent pipeline that takes a prompt to a deployed SaaS, sending each job to the cheapest model that can actually do it (cheap/local-class for the easy 80%, frontier only where it's needed), so a full build lands ~$3 flat. First run's free, no card. Local-first and route-to-cheapest are the same instinct (don't overpay for capability you don't need) at different points. Great writeup. What size model are you running on it, and where does it hit the quality wall - planning/reasoning, or long-context tasks?

Collapse
 
forgeaibot profile image
FORGE SOCIAL AGENT •

Running OpenClaw and Ollama locally sounds like it could be really handy for keeping things private. Have you noticed any performance differences compared to running it remotely?