DEV Community

Bruno Mello
Bruno Mello

Posted on Edited on

Running Local LLMs as Your AI Coding Assistant on Apple Silicon

#ai #programming #llm

Updated August 2026. The original version of this post ran Qwen3-Coder-30B on mlx_lm.server under mlx-lm 0.30.6. Eighteen months of daily use later, the architecture still holds, but almost everything I learned since then is about the things that only show up after you run this setup continuously: a still-unfixed Apple GPU driver bug, why your Mac gets slower as context fills, why benchmarks lie to you, and why the "easy" GUI option is not actually faster. Original content is preserved below; new material is marked [2026].

Tags: #llm #ai #programming #apple

What We're Building

The objective is to operate "OpenCode", an open-source agentic coding CLI, powered by a large language model running entirely on your Mac, without cloud dependencies, API costs, or external data transmission.

The Architecture (Three Layers)

Three interconnected components work together:

  1. OpenCode (The App) - User interface that sends messages and executes tools like file reading and code editing
  2. mlx_lm.server (The Hub) - Middleware translating between OpenCode and the model via HTTP on localhost:8080
  3. The LLM (The Brain) - Neural network processing numerical representations of text

Key Concepts

Tokens are numerical representations of text. Models process these numbers rather than raw text, with tokenizers handling conversion between text and token sequences.

Tool Calls represent structured requests where the model cannot directly access files but can request actions. The model outputs a tool call, the application executes it, and results feed back to the model for continued reasoning.

Tool Parsers interpret model-specific output formats. Different architectures use different structures (XML tags, JSON, special tokens), requiring appropriate parsers for proper function.

What Is MLX?

MLX is Apple's machine learning framework optimized for Apple Silicon. The key advantage: unified memory architecture allows the CPU and GPU to share RAM, enabling efficient model loading without copying overhead.

mlx-lm provides model loading, inference, and an OpenAI-compatible HTTP server through mlx_lm.server.

[2026] The Bug Nobody Warns You About: Kernel Panics from set_wired_limit

This is the most important thing in this post. If you take one thing away, take this.

If you run mlx_lm.server as a long-lived service, your Mac can kernel panic and reboot. Not crash the Python process: reboot the entire machine. The signature is:

panic(cpu 8 caller 0x...): "completeMemory() prepare count underflow" @IOGPUMemory.cpp:550
Kernel Extensions in backtrace:
   com.apple.iokit.IOGPUFamily(...)
Enter fullscreen mode Exit fullscreen mode

This is tracked as ml-explore/mlx#3186, open since March 2026. As of late August 2026 it is still unfixed, with no response from Apple or the MLX maintainers across the entire thread. It has been reproduced on M1 Ultra, M3 Ultra, M4 base, M4 Pro, M4 Max and M5, from 32GB to 256GB, so it is not a "you bought the wrong Mac" problem.

What actually triggers it

The valuable work in that thread is a controlled single-variable isolation by a commenter running ten fresh-server soak runs per arm (full report and shim). The findings overturned the obvious assumption:

  • It is not prefill size. A faithful replay of a crashing session, sequential requests with prefills up to 23k tokens, ran 111 minutes clean.
  • It is concurrency plus prompt-cache eviction churn. Add two concurrent request streams and unique prompts overflowing --prompt-cache-bytes, and it panics in 102 to 108 seconds, 3 out of 3 cold boots.
  • It is not OOM. Genuine memory exhaustion produces a clean userspace Metal error. The panic happens with headroom to spare.

The one mitigation that works

mlx_lm.server calls mx.set_wired_limit(...) at startup, raising the wired memory ceiling to roughly 75% of RAM. Wired pages cannot be reclaimed by the OS. That single call is the discriminating factor:

Change (one variable per arm) Result
none (stock) panic x3 @ 102 to 108s
set_wired_limit never called no panic, 10/10 runs, 4.2h
+ also no periodic clear_cache no panic, 10/10 runs, 5.0h
only skip periodic clear_cache panic @ 99.9s
sysctl iogpu.disable_wired_collector=1 + metal4 async mapping off panic @ 109.6s
RotatingKVCache / lower wired cap / MLX_MAX_OPS_PER_BUFFER die earlier as clean userspace OOM

Note what fails: the sysctl workarounds you will find on forums, and bounding the KV cache. Also worth knowing, --max-kv-size is silently ignored for architectures that define make_cache (the Qwen3.5 hybrids, for example).

The cost of the fix is about 3% sequential throughput, because weights become pageable.

Implementing it

There is no upstream opt-out. As of mlx-lm 0.31.3, mx.set_wired_limit is still called unconditionally in seven places, including mlx_lm/server.py. So you wrap the server in a launcher that no-ops it before importing:

#!/usr/bin/env python3
"""Launcher that disables set_wired_limit before mlx_lm.server starts.
Mitigation for ml-explore/mlx#3186."""
import os, sys
import mlx.core as mx

def main():
    if os.environ.get("MLX_LM_DISABLE_WIRED_LIMIT") == "1":
        mx.set_wired_limit = lambda *a, **k: 0
        print("[shim] wired limit disabled", file=sys.stderr)
    from mlx_lm.server import main as server_main
    sys.argv = ["mlx_lm.server"] + sys.argv[1:]
    server_main()

if __name__ == "__main__":
    main()
Enter fullscreen mode Exit fullscreen mode

Patching at the mlx.core module level matters: mlx_lm shares that module object, so every call site is covered, including the one inside server.py that a narrower patch would miss.

Run it with MLX_LM_DISABLE_WIRED_LIMIT=1 and confirm the [shim] line appears in your logs. Do not assume it applied.

[2026] Why Your Mac Gets Laggy as Context Fills

I lived with this for months without diagnosing it: the further a coding session got through a large context window, the more the whole machine dragged. Not just the model, everything.

It is the same wired memory, viewed from a different angle. As the KV cache grows, wired allocations grow with it, and wired pages cannot be reclaimed by macOS. So instead of the OS paging something out, your entire system starves.

With the shim in place, the same memory is pageable. Under a heavy multi-compaction session I watched free RAM dip to 44% and then recover to 86% once the task finished. That reclaim is exactly what wiring prevents. Same workload, no lag.

This is worth internalizing: that lag is not "the model is big", it is the early stage of the same unreclaimable-memory pathology that ends in the kernel panic above. If your Mac gets sluggish at high context, you are on the road to a reboot.

Setting Up

Prerequisites

  • Apple Silicon Mac (M1 or newer)
  • Minimum 32GB RAM (64GB+ recommended)
  • macOS with Homebrew

Installation Steps

Step 1: Install Python 3.12

brew install python@3.12
Enter fullscreen mode Exit fullscreen mode

Step 2: Create Virtual Environment

/opt/homebrew/bin/python3.12 -m venv ~/mlx-env
source ~/mlx-env/bin/activate
pip install --upgrade pip
pip install mlx-lm
Enter fullscreen mode Exit fullscreen mode

Step 3: Install OpenCode

npm install -g opencode-ai
# or
brew install anomalyco/tap/opencode
Enter fullscreen mode Exit fullscreen mode

Step 4: Start Model Server

source ~/mlx-env/bin/activate
mlx_lm.server --model mlx-community/Qwen3-Coder-30B-A3B-Instruct-8bit --port 8080
Enter fullscreen mode Exit fullscreen mode

Step 5: Configure OpenCode

Create ~/.config/opencode/opencode.json:

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "mlx": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "MLX (local)",
      "options": {
        "baseURL": "http://localhost:8080/v1"
      },
      "models": {
        "mlx-community/Qwen3-Coder-30B-A3B-Instruct-8bit": {
          "name": "Qwen3-Coder 30B 8bit",
          "tools": { "task": true }
        }
      }
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

[2026] Running It as a Real Service

Step 4 above is fine for trying things out, but a terminal-bound server dies with your terminal. Here is the LaunchAgent I actually use, at ~/Library/LaunchAgents/com.mlx-lm.coder.plist, wired through the shim:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>com.mlx-lm.coder</string>
    <key>ProgramArguments</key>
    <array>
        <string>/Users/you/mlx-env/bin/python3</string>
        <string>/Users/you/mlx-env/mlx_server_shim.py</string>
        <string>--model</string>
        <string>/Users/you/mlx-models/Qwen3.8-27B-MLX-4bit</string>
        <string>--port</string>
        <string>8081</string>
        <string>--chat-template-args</string>
        <string>{"enable_thinking": true}</string>
        <string>--prompt-cache-size</string>
        <string>4</string>
        <string>--prompt-cache-bytes</string>
        <string>8000000000</string>
    </array>
    <key>EnvironmentVariables</key>
    <dict>
        <key>MLX_LM_DISABLE_WIRED_LIMIT</key>
        <string>1</string>
    </dict>
    <key>RunAtLoad</key>
    <true/>
    <key>KeepAlive</key>
    <true/>
    <key>StandardOutPath</key>
    <string>/tmp/mlx-lm-coder.log</string>
    <key>StandardErrorPath</key>
    <string>/tmp/mlx-lm-coder.log</string>
</dict>
</plist>
Enter fullscreen mode Exit fullscreen mode

[2026] The launchd Trap That Cost Me an Hour

launchctl kickstart -k restarts the process but reuses launchd's cached job definition. Your edited plist is silently ignored.

The service comes back, the port answers, the logs look healthy, and your change did nothing. I edited that plist three times (added a token cap, changed concurrency, repointed --model to a new path) and none of it applied. I only noticed when the server failed to load after I moved the weights and the log showed it still opening the old path.

After any plist edit:

launchctl bootout gui/$(id -u)/com.mlx-lm.coder
launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.mlx-lm.coder.plist
Enter fullscreen mode Exit fullscreen mode

Then verify against the running process, never the file:

ps -o args= -p $(pgrep -f mlx_server_shim.py) | tr ' ' '\n'
Enter fullscreen mode Exit fullscreen mode

A corollary, since KeepAlive respawns a crashed server transparently: when you health-check a long-running service, compare the PID, not whether the port responds. A crash and respawn looks identical to healthy uptime from the outside.

[2026] Do Not Trust Your Own Benchmarks

I nearly published a completely fabricated result, and the mistake is easy to make.

I upgraded MLX, benchmarked, and measured a 2.7x speedup: 30 tok/s before, 84 tok/s after. Wonderful. Except a second model was resident on the GPU during the "before" run and not during the "after" run. With the GPU actually quiet, both versions do about 83 tok/s. The upgrade changed nothing.

On unified memory, a co-resident model does not just compete for RAM, it competes for memory bandwidth, which is the binding constraint for decode. That is enough to halve your numbers.

The tell is variance. A contaminated sweep produced 20.5, 36.6 and 72.4 tok/s for the same config. Wild run-to-run spread means contention, not noise. Clean runs are boring and tight.

So: before trusting any local LLM timing, confirm nothing else holds a model.

pgrep -fl mlx_server_shim.py   # or your equivalent
Enter fullscreen mode Exit fullscreen mode

And build the check into the harness so it aborts rather than producing a plausible wrong number.

[2026] Reading MLX Releases: Does This Commit Help You?

MLX ships fast, and release notes look exciting. Most of it will not touch your workload. A concrete example from 0.32.2:

  • "Read each K/V byte once in gqa-8 decode attention" looked like a direct win for my chat model. It does nothing for it. That path targets a GQA ratio of 8, and Gemma 4 26B-A4B is 16 attention heads over 8 KV heads, a ratio of 2. It never enters the optimized path.
  • "Fused full-attention path for head_dim 256" was even more tempting, since that model does have head_dim: 256. But it is gated to NAX devices, and an M3 Ultra reports applegpu_g15d.

Measured before and after on a quiet GPU: identical within noise.

The practical habit is to check the commit against your model's actual config.json before getting excited:

python3 -c "
import json; c=json.load(open('config.json')); t=c.get('text_config',c)
h,kv=t['num_attention_heads'],t['num_key_value_heads']
print('heads',h,'kv',kv,'GQA ratio',h//kv,'head_dim',t.get('head_dim'))"
Enter fullscreen mode Exit fullscreen mode

None of this means "skip the upgrade". I kept 0.32.2 because it is a large correctness release with no regression. It just is not a speedup, and I would rather say so than repeat a changelog.

[2026] LM Studio vs Direct mlx_lm.server

I ran the coder through LM Studio for a while, because JIT loading with an idle TTL is genuinely convenient. Then I moved it back to a direct mlx_lm.server. The measured difference on an identical agentic task:

Backend Same task, end to end
Direct mlx_lm.server 63.7s
LM Studio (MLX engine) 65.1s

A wash, and it should be: LM Studio's Apple Silicon engine is mlx-lm underneath. Same model, same quant, same speed. Anyone telling you one is dramatically faster than the other is measuring something else, probably contention (see above).

The real differences are operational:

Direct mlx_lm.server wins on:

  • Exact version control over mlx and mlx-lm
  • The wired-limit shim, which is the whole ballgame. LM Studio raises the wired limit and gives you no way to stop it
  • No cold start, since the model never idles out
  • Prompt cache that persists across sessions

LM Studio wins on:

  • JIT load plus idle TTL, so the model leaves RAM when unused. mlx_lm.server has no TTL and holds the model forever
  • A GUI, per-model load profiles, and reasoningEffort in the API

That last point matters if you migrate: reasoningEffort is an LM Studio API extension. mlx_lm.server ignores it. Set reasoning server-side instead, via --chat-template-args '{"enable_thinking": true}'. Check your model's chat_template.jinja for what it accepts. Qwen3.8 takes reasoning_effort of xhigh, medium or low, and raises on anything else.

My conclusion: use LM Studio to explore models, run production through mlx_lm.server with the shim.

[2026] OpenCode Harness Gotchas

Two behaviors that cost me real debugging time, because both look exactly like the model hanging:

  • opencode run blocks forever at init waiting on a permission prompt when there is no TTY. It never sends a single request to your server. Pass --auto.
  • It ignores the process working directory, resolving the project to $HOME and then failing its file picker with "Can not run certain FFF features in a file system root or home directories". Pass --dir <path>.

Both hangs reproduce identically against any backend, so a stall here is never evidence about your model or your server. Check whether a request actually arrived before you blame inference:

grep -c "POST /v1/chat/completions" /tmp/mlx-lm-coder.log
Enter fullscreen mode Exit fullscreen mode

The Model Hunt

(Original 2025 content, preserved. Current recommendation follows.)

Attempt 1: GLM-4.7-Flash-4bit

Failed because general-purpose models lack the precision needed for agentic coding workflows, producing repetitive loops on complex tasks.

Attempt 2: Qwen2.5-72B-Instruct-4bit

Downloaded successfully (~40GB) but processing took 18 minutes for modest context, causing connection crashes. Dense models activate every parameter per token, making them prohibitively slow despite fitting in RAM.

Attempt 3: Devstral-Small-2-24B-8bit

Server warning stated: "Received tools but model does not support tool calling." No available parser existed for this architecture in mlx-lm v0.30.6.

Attempt 4: Qwen3-Coder-30B-A3B-Instruct-8bit (Winner)

Successfully operated with correct tool parsing, strong instruction-following, and adequate speed. Uses approximately 33GB RAM. Mixture of Experts architecture activates only 3B of 30B parameters per token, achieving 3B-model inference speed with 30B-model knowledge.

Attempt 5: Qwen3-Coder-Next-4bit (Winner After Bug Fix)

The crash revealed a critical debugging lesson. The model shipped with "tool_parser_type": "json_tools" in its config, but outputs XML-formatted tool calls. The parser attempted JSON parsing on XML content, causing immediate failure.

The Fix: Manually edit the cached tokenizer_config.json:

find ~/.cache/huggingface/hub/models--mlx-community--Qwen3-Coder-Next-4bit \
  -name "tokenizer_config.json"
Enter fullscreen mode Exit fullscreen mode

Change "json_tools" to "qwen3_coder". After restart, tool calling functions properly.

[2026] Current Daily Driver: Qwen3.8-27B MLX 4-bit

Dense 27B, 256k native context, and it loads text-only under mlx_lm despite shipping a vision config (model_type: qwen3_5, with a text_config block). Roughly 15GB on disk, about 34 tok/s decode on an M3 Ultra.

Yes, dense, after I spent 2025 preaching MoE. The nuance I would add now: MoE still wins decisively on throughput per parameter, but a 27B dense model at 4-bit is fast enough for interactive agentic work while being noticeably more reliable at tool calling and long-horizon edits. For comparison, on the same machine a 26B-A4B MoE chat model runs about 83 tok/s, roughly 2.4x the tokens per second, and it is the right pick for conversation. For coding I will trade tok/s for fewer wrong turns.

Measured in real use: multiple compactions in a single session, prefills up to 146,838 tokens, 324 requests served, zero crashes. That last number is only possible because of the wired-limit shim. The original issue report in mlx#3186 was a panic on a ~173k-token prefill, which is the same territory.

Quantization Note

The community now publishes many quants of the same model at nearly the same size, and they are not equivalent. Measured against the bf16 original by top-1 token agreement, at ~16GB the various 4-bit builds land around 89%, while a 5.00bpw DWQ build at 17.7GB reaches ~91.5%, and 6-bit at 22.8GB reaches ~96.6%. If you have the RAM headroom, going one step up in bits buys more fidelity than switching between same-size 4-bit builds.

Treat vendor-published agreement charts with mild suspicion, though: DWQ methods are trained to reproduce the parent model's outputs, so "agreement with bf16" is a metric they optimize for directly. It is a fair number, not a neutral one.

Model Comparison Table

Model Type RAM Speed Tool Calls Verdict
GLM-4.7-Flash-4bit MoE ~16GB Fast Yes Loops on complex tasks
Qwen2.5-72B-4bit Dense ~42GB Very slow Untested Too slow for interactive use
Devstral-24B-8bit Dense ~25GB Medium No (no parser) Dead on arrival
Qwen3-Coder-30B-8bit MoE ~33GB Fast Yes Good 2025 pick
Qwen3-Coder-Next-4bit MoE ~50GB Fast Yes (after fix) Best quality (needs 64GB+ RAM)
Qwen3.8-27B-MLX-4bit Dense ~15GB ~34 tok/s Yes [2026] Current daily driver

Tool Call Flow

Complete sequence from user input to response:

  1. User enters query in OpenCode
  2. HTTP POST sends message and available tools to localhost:8080
  3. Server tokenizes message to token IDs
  4. Model runs inference on GPU
  5. Parser (e.g., qwen3_coder) converts XML output to structured JSON
  6. OpenCode receives parsed tool call and executes it
  7. Model receives execution results and generates natural language response

RAM and Model Selection Guidance

RAM Requirements:

  • 16GB: Small models only (7B-4bit), tight margins
  • 32GB: 7B-8bit or 13B-4bit comfortably
  • 64GB: Qwen3-Coder-30B-8bit (optimal)
  • 96GB+: Qwen3-Coder-Next-4bit or multiple models

Keep models under 75% of total RAM for system overhead.

[2026] A correction to my own advice above: "keep models under 75% of RAM" is where the danger actually lives, because ~75% is precisely what set_wired_limit claims by default. The rule is not wrong, but it is not the safety margin I implied. With wiring left enabled you can sit comfortably inside that budget and still panic the kernel, because the problem is unreclaimable memory, not insufficient memory. Disable the wired limit and the same budget behaves the way you expect.

MoE vs Dense Models:
Mixture of Experts architectures activate only a subset of parameters per token, providing speed comparable to smaller dense models while maintaining quality approaching their full parameter count. For local inference, MoE models deliver superior speed-to-quality ratios.

Pre-Download Checklist

  1. Verify mlx-lm has a tool parser for the architecture
  2. Confirm it's a coding-focused model
  3. Prefer MoE over dense architectures for interactive use
  4. Check quantization level against available RAM
  5. Validate tool_parser_type in the model's tokenizer_config.json matches actual output format
  6. [2026] Check num_attention_heads / num_key_value_heads if you care about specific kernel optimizations
  7. [2026] If it ships a vision config, confirm mlx_lm loads it text-only before committing to the download

Debugging Tips

  • Looping responses: Model may lack agentic coding capability; try alternatives
  • "Model does not support tool calling": No parser available for this architecture in current mlx-lm version
  • JSON decode errors on tool calls: Parser mismatch; verify tool_parser_type in config
  • Very slow responses: Dense model too large; switch to MoE or more quantized variant
  • Connection failures: Ensure server runs on correct port matching OpenCode configuration
  • [2026] Machine reboots under load: Not your fault. mlx#3186. Disable set_wired_limit
  • [2026] Whole Mac lags as context fills: Same root cause. Wired memory cannot be reclaimed
  • [2026] Config change had no effect: If it is a LaunchAgent, kickstart ignored your plist edit. Use bootout plus bootstrap
  • [2026] Benchmark numbers swing wildly: Another model is resident. Check before believing anything
  • [2026] opencode run hangs with no server request: Missing --auto or --dir, not a model problem

Conclusion

Running local AI coding assistants on Apple Silicon is practical and achievable, and eighteen months on I still use this daily instead of a cloud assistant.

What changed is my sense of where the difficulty lives. In 2025 I thought the hard part was model selection and tool parsers. Those are solved problems now. The hard part is everything that only appears once the thing runs continuously: an Apple driver bug nobody has fixed, memory that the OS cannot reclaim, benchmarks that lie when a second model is resident, and service tooling that silently ignores your edits.

None of that is in a quickstart guide, which is why it is in this one.

Resources

Top comments (3)

Collapse
 
rafal_boni_6af0797eadee3b profile image
Rafal Boni •

Thanks for this concise and helpful write-up... got it all working on my 36GB M3, although that machine is definitely too short of memory to run Qwen3.

However, as of mlx-lm v0.31.3 there is a Mistral parser, and it seems to work pretty well with Devstral-Small-2505-8bit, which is also small enough to run on the 36GB machine without blowing everything else out of the water (e.g. I can still run Chrome, the next biggest memory hog 🤦🏼‍♂️).

Collapse
 
lookingcloudy profile image
lookingcloudy •

Did you try olama?

Collapse
 
brunocerberus profile image
Bruno Mello •

Yes, Ollama is easier to configure, but it loses out on speed compared to MLX or LM Studio, as both are optimized for the Unified Memory Architecture on Apple Silicon