DEV Community: Michael

How to Run Local LLMs on Apple Silicon

Michael — Wed, 23 Sep 2026 07:18:45 +0000

This is about the model, on its own — what tool to install, what format to pull, and how to actually talk to it. Not wrapped in a VM, not wired up to a coding agent. If you want that part too, the complete AI agent guide covers it in depth; this page is the more basic thing underneath it.

Four names, four different things

These get used almost interchangeably in casual conversation, and they're not the same kind of tool:

llama.cpp is the inference engine underneath most of this ecosystem — a C++ implementation that runs GGUF-format models across NVIDIA, AMD, Intel, and plain CPU, not Apple-specific at all.
MLX is Apple's own array framework and inference library, built specifically for Apple Silicon's unified memory architecture, running its own MLX-format models.
Ollama is a friendly CLI and local API server wrapped around an inference backend — historically llama.cpp underneath, though that changed recently and specifically for Apple Silicon (more below).
LM Studio is a desktop GUI app that runs both GGUF and MLX models side by side in the same interface, and is the easiest on-ramp if you don't yet know which format or engine you want.

A real, recent change worth knowing about

As of Ollama 0.19 (March 2026), Ollama replaced its Metal-backed llama.cpp inference path with Apple's own MLX framework on Macs with 32GB or more of unified memory — a substantial speed improvement on supported hardware. 8GB and 16GB Macs still run the older Metal/llama.cpp path unchanged. If you're on a 32GB+ Mac, "MLX vs. Ollama" is a less clean-cut choice than it used to be: for many models, Ollama is now MLX under the hood, not a genuinely separate engine. Worth checking your installed Ollama version and hardware tier before assuming older comparisons still describe your setup exactly.

Model formats: GGUF vs. MLX, briefly

GGUF is llama.cpp's own model container format — broad, well-supported, with the longest tail of available conversions (older models, fine-tunes, merges) because the ecosystem around it is the largest. MLX-format models are built specifically for Apple's framework, and on the same chip an MLX build is commonly meaningfully faster than the equivalent GGUF for supported models. Neither format is strictly better — GGUF wins on breadth and portability, MLX wins on Apple Silicon-specific speed for models that have an MLX conversion available. Quantization — the 4-bit, 8-bit tradeoffs that actually determine how much RAM a given model needs — is its own topic: see LLM quantization explained for Mac users.

Where models actually come from

MLX-format models are published on Hugging Face, generally under the mlx-community organization; Ollama pulls from its own library. Both move fast — new releases and their MLX/GGUF conversions can lag each other by days to weeks, so treat any specific model tag you read (including on this blog) as a snapshot rather than a permanent recommendation, and check the source directly before pulling.

How much RAM you actually need

This blog already has a detailed RAM-to-model-size table, broken down from 16GB through 128GB+ Macs with specific MLX and Ollama tags per tier — see Section 3 of the complete AI agent guide rather than a second copy of the same table here. The short version: reserve roughly 20–25% of total unified memory for the OS and display compositor before budgeting the rest for model weights and KV cache, since both the model and macOS itself draw from the same pool. For the actual math behind those numbers — and how to work out a model that isn't already tabulated — see how much RAM local LLMs actually need.

Using a model with nothing else involved

None of this requires a VM or an agent. LM Studio gives you a chat window the moment a model finishes downloading. Ollama's own ollama run <model> drops you into a REPL directly in Terminal. Either one also exposes an OpenAI-compatible local API — http://127.0.0.1:11434/v1 for Ollama, whatever port you configure for MLX or LM Studio's server mode — that any script, editor plugin, or app that speaks the OpenAI API format can point at, with nothing more exotic than an environment variable pointing somewhere on your own machine.

When you'll actually want isolation

The model itself is inert — it's the code an agent generates and executes against that model that creates risk, and that's a different problem from "how do I run a model at all," worth its own guide rather than a section here. If what you're actually building is an AI coding agent, not just a chat window, see the complete AI agent guide for the full VM-isolated setup, the AI Bridge architecture for how the model stays on the host GPU while the agent runs isolated, and the MLX + OpenCode setup guide for one concrete, complete walkthrough.

Related reading: running AI agents safely in an isolated VM, the AI Bridge architecture, and what isolation actually costs, benchmarked across MLX and Ollama. Or download Velo Workspaces and try it yourself.

LLM Quantization Explained for Mac Users

Michael — Wed, 23 Sep 2026 07:16:01 +0000

The RAM math post treats quantization as an input — "4-bit is roughly 0.5 bytes per parameter" — and moves on, on purpose. This is what's actually behind that number: what quantization does to a model's weights, why "4-bit" isn't one single thing once you look at real GGUF filenames, and how to pick a level instead of copying whatever a guide happened to recommend.

What quantization actually is

A model's weights start out as 16-bit (or occasionally 32-bit) floating-point numbers — that's the precision they were trained and typically distributed in. Quantization replaces those with lower-precision representations: fewer bits per weight, grouped into blocks that share a small amount of extra data (a scale factor, sometimes a minimum value) so the reduced-precision numbers can still be mapped back to something close to the original range. Fewer bits per weight means less memory and less data to move through the chip per token generated — which is why quantized models are both smaller and faster, not just smaller.

The cost is precision, not correctness in any simple sense: a quantized model doesn't become "wrong" the way a training bug would make it wrong. It becomes a slightly blurrier version of the same model — individual outputs can differ from the full-precision original, more so at lower bit depths, but there's no bit width at which the model suddenly starts producing garbage across the board. The practical question is always "how much quality am I willing to trade for how much size and speed," not "is this broken."

Why "4-bit" isn't one thing

Open an actual GGUF download list and the names are more specific than "4-bit" or "8-bit" — Q4_K_M, Q5_K_M, Q4_0, Q8_0, and several more. The number is the average bits per weight; the letters describe the quantization scheme, and they're not interchangeable at the same bit count. The older, plain Q4_0/Q4_1 schemes quantize uniformly, block by block, with no per-layer nuance. The newer k-quants (Q4_K_M, Q5_K_M, and similar) are more deliberate about it — allocating more precision to weights that empirically matter more to output quality and less to weights that don't, rather than treating every block identically. At the same nominal bit width, a k-quant variant is close to universally the better choice over the legacy scheme it replaced; llama.cpp still ships both mainly for compatibility with older tooling.

The trailing letter on a k-quant (_S, _M, _L) is a small additional lever within that same bit width — small, medium, large — trading a bit more size for a bit more fidelity without changing the headline bit count. When two model cards both say "Q4," check the full tag before assuming they mean the same tradeoff.

Picking a level, not defaulting to one

A workable default rather than a rule: Q4_K_M is the level most of the ecosystem converges on as the practical sweet spot — a meaningful size and speed win over 8-bit, with a quality gap that's genuinely hard to notice in normal use for most tasks. Move up from there deliberately, not reflexively:

Q5_K_M or Q6_K when the task is precision-sensitive — code generation and anything involving exact syntax or careful multi-step reasoning tends to show quantization loss sooner than open-ended writing does.
Q8_0, close to full precision, when you have the memory headroom to spare and want the smallest possible gap from the unquantized model, accepting the larger download and footprint.
Q3_K or lower only when memory is the hard constraint and running a smaller parameter count at a higher bit level isn't an option — a heavily quantized larger model is not reliably better than a lightly quantized smaller one at the same memory budget, and often worse. Test both rather than assuming bigger-but-blurrier wins. MLX-format models use their own bit-depth conventions rather than the GGUF/k-quant naming above, but the underlying tradeoff — bits per weight traded against quality — is the same idea wearing a different label. The tools and formats post covers where each format actually comes from.

What this changes about the RAM math

Going up one quantization tier doesn't just cost more memory for weights — it changes the whole budget covered in the RAM requirements post: weights scale directly with bits per parameter, and KV cache size is independent of quantization level entirely, computed from the model's architecture rather than its weight precision. A model that just barely fits at Q4_K_M with room for a long agentic session may not have any KV cache headroom left at all at Q8_0, even though the weights themselves are still well within memory. Check both numbers before moving up a tier, not just whether the weights alone fit.

Related reading: how much RAM local LLMs actually need, the math this post's bit-width numbers feed into, and the local LLM tools and formats landscape for where to actually find quantized models. Or download Velo Workspaces and try it yourself.

How Much RAM Do You Need for Local LLMs on a Mac?

Michael — Wed, 23 Sep 2026 07:15:00 +0000

This blog already has a tier-by-tier table for the specific models it recommends. This is the math behind it — so you can work out the answer for any model, not just the ones already tabulated, and understand why a model that fit fine when you started a session can run out of room an hour later.

Two numbers, not one

A model's memory footprint isn't a single static figure — it's two genuinely different things added together: the weights, which are fixed the moment you pick a model and a quantization level, and the KV cache, which grows as a conversation gets longer. Sizing for weights alone is the mistake that makes a model work fine in a quick test and then run out of memory in a real session.

Weights: the simple part

Weight memory is roughly parameter count times bytes-per-parameter. A model's name tells you the first number — "7B" means 7 billion parameters — and quantization tells you the second: 4-bit quantization is roughly 0.5 bytes per parameter, so a 7B model at 4-bit lands around 3.5GB just for weights. Double the bit-width and you roughly double that figure. (For what's actually behind that bytes-per-parameter number — the different quantization schemes, and why they're not interchangeable at the same bit width — see LLM quantization explained for Mac users.) This is the part every model card and download page already tells you, directly or indirectly, and it's the part most "how much RAM" answers stop at — which is exactly the gap that causes surprises later.

KV cache: the part that grows on you

The key-value cache stores intermediate attention state for every token in the current context, and unlike weights, it scales with how long the conversation gets. The actual formula: 2 × layers × KV-heads × head-dimension × tokens × bytes-per-element — each of those numbers is in the model's own config, and most inference tools will report the resulting figure directly rather than making you compute it by hand. The practical upshot is what matters here: for a short exchange, weights dominate and KV cache is nearly irrelevant. For a long agentic session with a large context window — exactly the kind of multi-turn, tool-calling conversation an AI coding agent produces — KV cache can grow to rival or exceed the weights themselves. That's the mechanism behind "it worked when I started, then got slow or crashed later": nothing changed about the model, the context just kept growing.

The other consumer: macOS itself

Unified memory means the model isn't the only thing drawing from the pool — the OS and display compositor need their share too, which is why the existing model-sizing table reserves roughly 20–25% of total RAM for that overhead before budgeting the rest. That reserve isn't a made-up safety margin; it's covering real, measured system usage that doesn't show up in a naive "weights + KV cache" calculation.

Putting it together

A workable estimate for any model, not just the ones already tabulated on this blog: take the parameter count times bytes-per-parameter for weights, add a KV cache estimate scaled to how long a session you actually expect (short chat: small; long agentic session: meaningfully more), then make sure the total leaves that 20–25% headroom for macOS. If a model's weights alone already eat most of your unified memory, there's no room left for KV cache to grow into — that's the model to size down, not the one to fight with swap.

What it looks like when you get it wrong

Memory pressure on a Mac doesn't usually announce itself as a hard crash — it shows up as swapping, and an inference engine that was fast a minute ago suddenly isn't, with no error message pointing at the actual cause. If a model that felt snappy at the start of a session gets progressively slower rather than staying consistent, check context length and KV cache growth before assuming something's wrong with the model or the engine itself.

Related reading: the tier-by-tier model table this post explains the math behind, the broader landscape of local LLM tools, and how much memory to give a Linux VM for non-LLM workloads, including the memory-balloon device that handles the VM side of memory allocation. Or download Velo Workspaces and try it yourself.

Zero-Tax Virtualization: Running AI Agents Safely in Velo Workspaces

Michael — Wed, 16 Sep 2026 17:24:00 +0000

1. Introduction: The Security vs. Performance Dilemma

Running autonomous AI coding agents natively on your primary macOS machine introduces severe security liabilities. Modern agents execute terminal commands, install unverified dependencies via pip and npm, modify arbitrary files, and can be coerced through prompt injection attacks into exfiltrating environment variables, dotfiles, or macOS Keychain secrets.

The standard industry remedy is sandboxing agents inside a Linux virtual machine or container. On Apple Silicon, however, this immediately collides with a hypervisor-level barrier:

No Compute Passthrough: Apple's native Virtualization.framework does not expose the host Metal GPU to Linux guests. Linux VMs receive only a 2D paravirtualized framebuffer (virtio-gpu).
The "Virtualization Tax": Attempting to run an LLM directly inside a Linux VM forces model inference onto virtualized CPU cores. Generation speeds crater by 80%+, making multi-turn agentic loops unusable.
Containers Share the Same Limitation: Docker Desktop, OrbStack, and Colima run atop Linux guest kernels and face the exact same GPU compute void.

The Decoupled Solution: Velo Workspaces AI Bridge

Velo Workspaces resolves this trade-off by separating the agent execution environment from the model inference engine:

Model on the Host: An inference server — Ollama or Apple MLX — runs natively on macOS, retaining full access to unified memory bandwidth and Metal GPU acceleration.
Agent in the VM: The agent framework executes inside an isolated Ubuntu Linux guest, restricting all file modifications and terminal execution to a sandboxed filesystem.
VirtIO-vsock Transport: Traffic travels across hypervisor memory buffers via vsock rather than a traditional virtualized NAT network stack, reducing bridge overhead to single-digit milliseconds.

This guide covers both engines side by side. Pick one in Section 5 — everything downstream (VM setup, the vsock bridge, agent configuration) works identically either way, substituting the port your chosen engine listens on.

	MLX	Ollama
Default port	`8080`	`11434`
Model source	huggingface.co/models?library=mlx-lm	ollama.com/library
Best for	Apple Silicon-native performance, the widest current selection of day-one MLX-quantized releases	The simplest one-command setup and model management (`ollama pull`, `ollama run`)

2. Architecture Topology

Note: Inside the VM, the agent's traffic is forwarded over a VirtIO-vsock channel to the AI Bridge on the macOS host, which relays it to the host's inference server (Ollama or MLX). A separate Caddy port forward lets an external LAN client reach the VM's optional web UI.

Network Flow Breakdown

Inference Path: The agent sends standard OpenAI-compatible HTTP requests to 127.0.0.1:<PORT> inside the VM — 8080 for MLX, 11434 for Ollama. The local socat proxy routes this payload across vsock to the host. The Swift AI Bridge receives the connection and relays it to whichever engine you selected as the workspace's Host Provider.
Host UI Access: The host browser connects directly to the guest's virtual interface via http://<VM_IP>:4096 over the standard hypervisor bridge.
External LAN Access: Because external machines cannot route directly to the VM's private virtual subnet, the host forwards an external port (8081) to the guest's Web UI port (4096). Port 8081 is used to avoid colliding with whichever inference port (8080 or 11434) the host is already using.

3. Choose Your Model (Purpose and Host RAM Requirements)

Both engines use macOS Unified Memory dynamically. Because the OS, display compositor, and the model's KV cache share this pool, reserve at least 20–25% of total host RAM for operating overhead.

Model identifiers below are verified as of this writing — always confirm current availability and exact tags at huggingface.co/models?library=mlx-lm (MLX) or ollama.com/library (Ollama) before pulling, since libraries change.

Mac Unified RAM	MLX (Hugging Face)	Ollama (`ollama pull ...`)	Quantization	Use Case
16 GB	`mlx-community/Qwen2.5-Coder-7B-Instruct-4bit`	`qwen2.5-coder:7b`	4-bit	Fast code completion, lightweight script generation, single-file edits.
24 GB / 32 GB	`mlx-community/Qwen2.5-Coder-32B-Instruct-4bit` `mlx-community/Mistral-Small-24B-Instruct-4bit`	`qwen2.5-coder:32b` `mistral-small`	4-bit	Multi-file reasoning, refactoring, and debugging complex logic.
36 GB / 48 GB	`mlx-community/Qwen3.8-27B-4bit`*	`qwen3.8:27b`*	4-bit	Advanced agentic tasks, architectural design, repository-wide indexing.
64 GB / 96 GB	`mlx-community/Llama-3.3-70B-Instruct-4bit`	`llama3.3:70b`	4-bit	Deep reasoning, zero-shot full repository synthesis, complex planning.
128 GB+	`mlx-community/Qwen3.8-2.4T-A95B-bit` `deepseek-ai/DeepSeek-V3`*	`deepseek-v3`*	1-bit to 4-bit	Full-scale autonomous pipelines, heavy concurrent agent swarms.

* These are very recent (2026) or very large releases — check the library link for the exact current tag/quantization before pulling; MLX and Ollama conversions can lag a new release by days to weeks.

4. VM Sizing Guidelines

Because the heavy LLM weights and KV caches stay in macOS unified memory, the Linux VM only needs enough resources to execute the code generated by the agent.

Important (Session Longevity): Long-running autonomous sessions gradually leak resources. Agents continuously generate temporary files, compile dependencies, bloat pip/npm caches, and retain execution logs. Allocate extra memory buffers for long-lived environments to keep the Linux Out-Of-Memory (OOM) killer from terminating tasks.

Scenario	vCPUs	Memory	Storage	Primary Workloads & Rationale
Ephemeral / Light Automation	2 vCPUs	2 GB – 3 GB	15 GB – 20 GB	CLI automation and simple scripts. Ideal for short-lived, disposable tasks.
Full-Stack Web Development	4 vCPUs	4 GB – 6 GB	30 GB – 40 GB	Node.js, Django, SQLite, Vite. Accommodates build tools and background servers.
Long-Running Agent Sessions	4 – 6 vCPUs	8 GB – 12 GB	50 GB – 60 GB	Continuous autonomous loops. Buffers memory against cached artifacts and logs.
System Programming & Docker	6 – 8 vCPUs	12 GB – 16 GB	60 GB – 80 GB	Rust/Go/C++ compilation, Docker-in-VM services. Prevents compilation lockups.

5. Install the Inference Engine on the macOS Host

Pick one engine. All steps in this section run in the macOS Terminal on the host.

5.1 Option A — MLX

Install the official Apple MLX language model package using Python (3.10+):

# Create and activate an isolated virtual environment
python3 -m venv ~/.mlx-env
source ~/.mlx-env/bin/activate

# Install the MLX LM server package
pip install --upgrade mlx-lm

Launch the server bound to loopback (127.0.0.1) on port 8080. It downloads the model from Hugging Face automatically on first run. mlx-community/Qwen2.5-Coder-7B-Instruct-4bit below is just an example — swap in whichever tag you picked for your RAM tier in Section 3:

mlx_lm.server \
  --model mlx-community/Qwen2.5-Coder-7B-Instruct-4bit \
  --host 127.0.0.1 \
  --port 8080

In a second macOS terminal, verify it's responding to OpenAI-compatible requests:

curl -s http://127.0.0.1:8080/v1/models | grep "id"

5.2 Option B — Ollama

Install Ollama from ollama.com (or brew install ollama). qwen2.5-coder:7b below is just an example — pull whichever tag you picked for your RAM tier in Section 3 instead:

brew install ollama
ollama serve &          # or just launch the Ollama app — it runs this for you
ollama pull qwen2.5-coder:7b

Ollama listens on 127.0.0.1:11434 by default. Verify it's responding:

curl -s http://127.0.0.1:11434/v1/models | grep "id"

6. Create a VM in Velo Workspaces

Launch Velo Workspaces on macOS.
Choose AI Workspace in the sidebar, click + in the upper right.
Pick an image source: download an official image, choose a local OS image file, or boot from an existing virtual disk. Let's pick Ubuntu 26.04 LTS (Server or Desktop).
Choose the AI Sandbox profile, click "Show Advanced Settings".
Assign hardware parameters based on the sizing guidelines in Section 4 (e.g., 4 vCPUs, 6 GB RAM, 40 GB Disk). Ensure AI Bridge is checked and Host Provider matches whichever engine you started in Section 5 — MLX or Ollama.

Example: AI Sandbox profile with AI Bridge enabled and Host Provider set to MLX. Pick Ollama here instead if that's what you started in Section 5.

Click Continue, install the guest OS, then start the virtual machine and log in.

7. Install and Configure AI Agents inside the VM

All steps in this section run in the Ubuntu Linux Terminal inside the VM. Throughout, <PORT> is 8080 for MLX or 11434 for Ollama — whichever you picked in Section 5.

7.1 Establish the Guest Vsock Proxy

Rather than typing this by hand, open the running workspace's AI Bridge tab in Velo Workspaces — it shows your exact port already filled in, with a Copy button on each command block. Run step 1, then either step 2 (forwards for the life of this terminal session — simplest, good for a quick test) or step 3 (installs it as a systemd service that survives reboots — better for anything you'll come back to):

# 1. Install socat
sudo apt-get install -y socat

# 2. Forward the port for this session
socat -d -d TCP-LISTEN:<PORT>,fork,reuseaddr,bind=127.0.0.1,nodelay VSOCK-CONNECT:2:<PORT>

— or, to keep it running across reboots —

# 3. Install as a systemd service instead of step 2
sudo tee /etc/systemd/system/velo-ai-bridge.service >/dev/null <<'EOF'
[Unit]
Description=Velo Workspaces AI Bridge (127.0.0.1:<PORT> to the host over the high speed channel)
After=network.target

[Service]
ExecStart=/usr/bin/socat TCP-LISTEN:<PORT>,fork,reuseaddr,bind=127.0.0.1,nodelay VSOCK-CONNECT:2:<PORT>
Restart=always
RestartSec=2

[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable velo-ai-bridge
sudo systemctl restart velo-ai-bridge

Then, either way, verify guest-to-host connectivity across the vsock boundary:

# 4. Check it from inside the workspace
curl -sS http://127.0.0.1:<PORT>/v1/models

If that returns your model's info, the bridge is live — every agent below points at http://127.0.0.1:<PORT>/v1, and each one configures that itself in the next section, so there's no separate global environment-variable setup needed here.

7.2 Configure Agent Frameworks

Agent Option A: OpenCode (CLI & Web UI)

OpenCode provides both an automated CLI agent and an interactive web workspace.

Install OpenCode and its build dependencies:

   sudo apt install -y curl git build-essential
   curl -fsSL https://opencode.ai/install | bash
   source ~/.bashrc

Register your inference engine as a provider. Create ~/.config/opencode/opencode.json:

   mkdir -p ~/.config/opencode
   nano ~/.config/opencode/opencode.json

Paste this, substituting <PORT> and the model name for your chosen engine (from Section 3):

   {
     "$schema": "https://opencode.ai/config.json",
     "provider": {
       "local": {
         "npm": "@ai-sdk/openai-compatible",
         "name": "Local Server",
         "options": {
           "baseURL": "http://127.0.0.1:<PORT>/v1"
         },
         "models": {
           "<model-name>": {
             "name": "<Display Name>"
           }
         }
       }
     },
     "model": "local/<model-name>"
   }

Run in non-interactive CLI mode — --auto auto-approves tool execution inside the sandbox:

   opencode run --auto "Write a python script to benchmark disk I/O, execute it, and print the results."

Or launch the interactive TUI, then connect via its wizard instead of (or in addition to) the config file:

   opencode

Inside the TUI, type /connect, select Local Server, and when prompted for an API key, type anything (e.g. local) — the local server doesn't check it.

Or run the web interface, bound to 0.0.0.0 so it's reachable across the hypervisor network:

   opencode web --port 4096 --hostname 0.0.0.0

From your Mac's browser: http://<VM_IP>:4096.

Accessing the Web UI from another PC on your LAN: external machines can't route directly into the VM's private subnet, so forward a port on the host. On the macOS host:

   brew install caddy
   caddy reverse-proxy --from :8081 --to <VM_IP>:4096

Any machine on the LAN can then navigate to http://<HOST_IP>:8081.

Agent Option B: Open Interpreter

Open Interpreter provides a direct terminal agent loop designed for code execution.

Install:

   pip install open-interpreter

Launch against the local endpoint — -y auto-approves code execution without a confirmation prompt each time:

   interpreter \
     --api_base http://127.0.0.1:<PORT>/v1 \
     --model <model-name> \
     --api_key local \
     -y

Agent Option C: Aider

Aider is designed specifically for Git-integrated pair programming and repository modifications. It reads the endpoint from environment variables rather than dedicated CLI flags, and needs the openai/ prefix on the model name so it routes through its OpenAI-compatible path:

Install:

   python3 -m pip install aider-chat

Run inside a Git repository:

   cd /path/to/project
   export OPENAI_API_BASE=http://127.0.0.1:<PORT>/v1
   export OPENAI_API_KEY=local
   aider --model openai/<model-name>

Agent Option D: Goose

Goose is an extensible open-source autonomous agent developed by Block.

Install:

   curl -fsSL https://github.com/block/goose/releases/download/stable/download_cli.sh | CONFIGURE=false bash
   export PATH="$HOME/.local/bin:$PATH"

Configure a custom OpenAI-compatible provider:

   goose configure
   # Select: Add Provider → Custom / OpenAI-compatible
   # Base URL: http://127.0.0.1:<PORT>/v1
   # API Key:  local
   # Model:    <model-name>

Start a one-shot autonomous run:


bash
   goose run --text "Audit this directory, find security misconfigurations in JSON files, and correct them."

An Architecture to Run AI Agents Safely and Efficiently in a Linux VM on Apple Silicon

Michael — Wed, 09 Sep 2026 14:10:23 +0000

I'm the developer of Velo Workspaces, a native macOS app for disposable Linux and macOS VMs on Apple Silicon. This post is about a specific problem I kept hitting while building it, how to let an AI agent run generated code safely in a VM without giving up the GPU acceleration? I ended up with the architecture AI Bridge based on vsock, and supported by real benchmark numbers.

The problem

If you want to run an AI coding agent locally on your Mac today, you may choose one of the following two options.

Option A: run it directly on your Mac. Most agent frameworks, like LangChain, AutoGen, SWE-Agent and Open Interpreter, execute LLM-generated code via subprocess or exec() with full access to your filesystem, network, and credentials. Docker's own engineering blog has documented real AI coding agent security incidents resulting from exactly this. A recent dev.to post lays out the same concern plainly: AI Agents Run Unsandboxed Code — How to Fix It. Indirect prompt injection makes this worse than it sounds, and a malicious instruction hidden in a file the agent reads can trigger commands with your full permissions, not just the ones you asked for.

Option B: run it in a VM or container for isolation. This is the "correct" answer on paper, and it runs straight into a wall specific to Apple Silicon: Apple's Virtualization.framework does not expose the host GPU to a Linux guest like the way it does for macOS guests. This isn't a rumor, it's confirmed directly by Apple's own container team in their public repos, in response to people asking for exactly this: GPU passthrough availability? and Paravirtualized Graphics via virtio-gpu. A macOS guest gets a purpose-built Metal-based GPU driver; a Linux guest gets virtio-gpu, a paravirtualized 2D device with no path to the host's compute GPU. Running a 7B model inside the VM is dramatically slower than the same model on the host.

Root cause: on Apple Silicon, "safe" (inside a VM) and "fast" (on the GPU) currently pull in opposite directions, because the GPU only exists on the host side in the case of Linux VMs.

The solution: split the agent from the model

The fix isn't to pick a side, it's to stop treating "run the agent" and "run the model" as a same problem. The dangerous part (executing LLM-generated code) and the expensive part (running the model) don't have to live in the same place:

The agent runs inside the isolated, disposable Linux VM, and when you're done, the whole VM (and anything it wrote to disk) is gone.
The model runs on the host, on Ollama or MLX, with full native Metal GPU access.
AI Bridge connects the two over a direct vsock channel, so the agent inside the VM talks to the model exactly like it would talk to any local server.

┌───────────────────────────────────────────────────────────────────┐
│                            macOS HOST                             │
│                                                                   │
│      Ollama / MLX Engine   ───►  Storage: ~/.ollama/models        │
│      (Native Metal GPU)          (single 10GB–50GB copy)          │
│              ▲                                                    │
│              │  Velo Workspaces AI Bridge                         │
│              │  proxies to localhost:<port>                       │
│              ▼                                                    │
│      ┌─────────────────────────────────────────────────────┐      │
│      │                     LINUX VM                        │      │
│      │  socat 127.0.0.1:<port>  ⇄  vsock CID 2             │      │
│      │  Env: OPENAI_API_BASE=http://127.0.0.1:<port>/v1    │      │
│      │  Dev apps / Docker containers / VS Code / Python    │      │
│      └─────────────────────────────────────────────────────┘      │
└───────────────────────────────────────────────────────────────────┘

A relay bridges a local TCP port inside the guest to the host's model-server port over vsock (CID 2 is the host, in Apple's vsock addressing). The port isn't hardcoded to Ollama, so the bridge works for MLX or any other OpenAI-compatible local server on whatever port it's actually listening on. From inside the VM, you set OPENAI_API_BASE (or whatever your tool calls it) to http://127.0.0.1:<port>/v1 and every OpenAI-compatible client will work, such as Open Interpreter, LangChain, or a raw curl. No code changes need to be done, and no awareness that a VM boundary exists at all.

The value this targets specifically: get real isolation for the part that's actually risky (arbitrary generated code execution) without paying a GPU-virtualization tax for the part that's actually expensive (inference), and without copying a 10–50GB model file into every VM you spin up.

The results

I wanted to know how close this actually gets to "no tax" so I benchmarked it properly rather than asserting it.

Setup:

Host: Mac mini, Apple M4, 10 cores, 16GB RAM, macOS 26.6.2.
Guest: Ubuntu Server 26.04 (minimized), 4 vCPU, 4GB RAM.
Model: qwen2.5-coder:7b (Q4_K_M) served by Ollama on the host.

Bridge tuned with a larger socat buffer and nodelay set. Both sides on Python 3.12.

Three tests, scripts here:

ai_perf_test.py — single-request time-to-first-token and throughput, 5 runs, deterministic
ai_load_test.py — 8 concurrent clients (OLLAMA_NUM_PARALLEL=2)
oi_perf_test.py — full agentic-loop latency via Open Interpreter: prompt → generated code → execution → result, 5 runs × 3 tasks

One methodology lesson that mattered more than anything else: the first time I ran these, I switched between the host test and the VM test without being disciplined about fully stopping the VM and restarting Ollama between them. The agentic-loop numbers were a mess on host, wildly bimodal, some iterations 10x slower than others. The fix was to stop all VMs, stop Ollama, restart ollama serve fresh, before every single run for both host and VM tests. Once I did that consistently, the bimodal pattern disappeared completely. A leftover process from the previous run contending for CPU/scheduling with the one actually being measured was enough to produce numbers that looked dramatic but meant nothing. All results below are from the controlled re-run.

Raw single-request speed — this is the actual "tax," and it's small

	Host (loopback)	VM (AI Bridge)	Delta
Time to first token	56 ms	59 ms	+6%
Throughput	22.09 tok/s	21.78 tok/s	−1%

These numbers reveal the actual cost of this setup: exactly one network hop over vsock. Because the model stays firmly anchored to the host's hardware, a single-digit percentage drop is practically as close to a "zero virtualization tax" as physics will allow. You aren't paying a penalty for slow GPU emulation simply because there is no GPU emulation happening at all.

8 concurrent clients — the VM won

	Host (loopback)	VM (AI Bridge)
Total wall time (8 clients)	15.04s	12.73s
Avg per-client time	9.43s	8.28s

The VM finished ~15% faster via the bridge！ Initially, I thought this was purely about CPU isolation on the host, the benchmark client and the Ollama server are fighting for the exact same physical cores. In the VM, the client gets its own dedicated vCPU slice. But digging into the network architecture reveals a deeper mechanical advantage. When you run 8 concurrent Python clients natively on macOS, Python relies on kqueue for async I/O. It works, but it isn't quite as efficient as Linux's epoll event loop for handling rapid socket creation. More importantly, when the VM routes traffic over vsock, it lands in Velo Workspaces's native Swift app on the host, which uses Apple's Grand Central Dispatch (GCD) to proxy the connection to Ollama. GCD acts like a highly optimized shock-absorber at the Apple Silicon hardware level. It batches and streams those requests to localhost far more efficiently than a raw Python script hammering the port directly. You are effectively putting an enterprise-grade reverse proxy in front of the model.

Real agentic loop — the VM won consistently, by a believable margin

Task	Host mean	Host std dev	VM mean	VM std dev
Math computation	6.45s	±1.16s	5.38s	±1.28s
System info	7.14s	±1.12s	5.82s	±1.07s
File I/O	9.08s	±1.22s	6.56s	±1.38s

The VM ran 17–28% faster, with both sides showing tight, comparable variance and no bimodal weirdness once background contention was controlled for. Seeing the VM consistently beat native macOS on tasks like File I/O and System Info might look counter-intuitive, but it makes perfect sense when you look at how agent frameworks actually execute code. Open Interpreter relies heavily on spawning background terminal shells and Python REPLs, writing temporary files, and reading stdout. Every time a script does this natively on macOS, Apple's background security daemons (XProtect, Gatekeeper, and Endpoint Security) briefly intercept the execution to scan for malicious behavior. It is a tiny delay that compounds massively over the course of an agentic loop. Inside the minimized Ubuntu VM, none of that desktop antivirus overhead exists. Linux's process creation (fork()) is fundamentally lighter than macOS's (posix_spawn), and writing to a raw ext4 filesystem is instantaneous. The vsock bridge penalty is so remarkably small that it gets completely swallowed by the sheer execution speed of Linux.

How close to the ideal outcome?

The goal was "near without virtualization tax" — not literally zero, since a network hop is unavoidable, but close enough that it isn't the thing you'd notice. A single-digit-percent difference on the raw inference path is that outcome. What I didn't expect going in was that the system-level numbers — concurrent load, real agentic workflows — would come out ahead in the VM rather than merely close. The isolation you get for safety turns out to also isolate you from host-side resource contention, which is a genuine bonus on top of the safety case, not just a wash.

Biggest lesson overall: environment hygiene between benchmark runs matters as much as the architecture you're testing. A leftover process from the previous run can produce numbers dramatic enough to completely mislead you about which side actually won.

Scripts are at github.com/xgz2025/veloworkspace-tools if you want to reproduce this on your own hardware or model. Velo Workspaces, which implements this architecture as AI Bridge, is at veloworkspaces.com.