DEV Community: akartit

I Ran Qwen3.8-27B on a 24GB MacBook - Here Is the VRAM Number Apple Does Not Print

akartit — Thu, 27 Aug 2026 21:39:47 +0000

I tried to run a 27B parameter model on a 24GB MacBook.

It kernel-panicked my Mac. Three times. Hard reboot, lost work, the whole thing. And the fix every forum gives you made it worse.

Here is the number nobody prints, and how to find yours.

Click to watch: 12 minutes, the three panics, picking the quant, and the model driving a TV over MCP.

Your Mac has 24GB. Your GPU gets 17.76 GiB.

That is about 74% of what is printed on the box. Your weights, your KV cache and your compute buffers all share it.

Every guide comparing a GGUF's file size against your total RAM is doing arithmetic against a number the GPU never sees.

Find yours right now:

ioreg -l | grep recommendedMaxWorkingSetSize

Divide by 2^30.

Your Mac	Expected GPU working set
16 GB	~11.8 GiB
24 GB	17.76 GiB (measured)
36 GB	~26.6 GiB
48 GB	~35.5 GiB
64 GB	~47.4 GiB

Only the 24GB row is measured on my machine. Run the command and trust your own number.

Worth being precise about what this is, because the popular version is wrong. Apple does not call it a cap. Their own definition is:

An approximation of how much memory, in bytes, this GPU device can allocate without affecting its runtime performance.

It is a threshold Apple recommends you stay under, not a wall that stops you. Nothing refuses your allocation at 17.76 GiB. You are free to walk straight past it, into territory Apple told you to avoid, which on this machine contains a driver bug.

Why it panics instead of erroring

Q4_K_M, the quant everyone recommends, is 15.3GB of weights and peaks at 19.6GB once the projector and runtime buffers load. That is 110% of the budget.

panic(cpu 9): "pending memory object unexpectedly found in non pending hash"
IOGPUGroupMemory.cpp:528
process: WindowServer

The important line is elsewhere in the log:

Compressor Info: 4% of compressed pages limit (OK)

Four percent. The machine did not run out of memory. It hit a bug in Apple's GPU driver.

Not just me, and not just llama.cpp. The same failure is filed repeatedly against MLX: a kernel panic on an M4 Max during a large prefill, and unbounded KV cache growth taking down a Mac Studio. That thread names the mechanism: the memory is wired, so macOS's out-of-memory killer cannot reclaim it and never even sees pressure.

A normal process that asks for too much gets killed. A GPU allocation that asks for too much takes the kernel with it.

Why raising the wired limit backfires

Every forum says sudo sysctl iogpu.wired_limit_mb=20480. It is backwards.

Normal memory is a car in a car park. If the system needs the space, it tows the car.

Wired memory is bolted to the ground. The kernel is not allowed to move it, compress it, or swap it out. That is what the word means.

Raising the limit does not make the car park bigger. It bolts down more cars. You get less room to shuffle, and you remove macOS's ability to dig itself out.

I raised it. It panicked again.

"But I have 64GB and it is still slow"

This is the most common reply I get, so it gets its own answer.

RAM decides what fits. Bandwidth decides how fast. Separate limits, constantly conflated.

Your GPU re-reads the whole active weight set for every token it writes, so speed is roughly bandwidth divided by model size. Your RAM figure tells you nothing about your bandwidth:

Chip	Bandwidth
M4 Pro	273 GB/s
M2 / M3 Max	400 GB/s
Ultra	800 GB/s

Three times the spread at the same 64GB. On the Pro tier you are capped and no setting fixes it. You are likely comparing yourself to people with double your bandwidth.

What actually works

llama-server -m Qwen3.8-27B-UD-Q3_K_XL.gguf \
  --mmproj mmproj-F16.gguf \
  -ngl 99 -c 16384 -fa on -ctk q4_0 -ctv q4_0 --jinja

-ctk q4_0 -ctv q4_0 quantises the KV cache itself to 4 bits, roughly 4x smaller. On this machine it is the difference between 16k context fitting and not. Quantise your cache before you drop a tier on your weights. You lose far less.


Decode, clean machine	12.60 tok/s
Same config, 4.15GB of swap present	11.26 tok/s
Prefix cache	14.2x (47.9s to 3.4s on 4,826 tokens)
Objective tasks	21/21
Tool calling	6/6
Wired memory	16.37 GiB, 92% of budget

That second row is the point. Same everything, 11% slower purely because the machine was carrying swap from a long session. Eleven percent of your throughput can be sitting in a browser tab you forgot about.

And one thing that nearly caught me out: my fit tool estimated 90% of budget. Measured, it wires 92%, hitting 94% the moment you show it an image. Treat any fit estimate as a floor, not a ceiling.

Will it fit on your machine

pip install localfit
localfit qwen3.8-27b --gpu-gb 12

--gpu-gb scores every quant against your budget instead of mine. Feed it your own ioreg number.

AnassKartit / localfit

Will it fit? GPU toolkit for AI models — MLX + GGUF + Cloud in one menu.

localfit

██╗      ██████╗  ██████╗ █████╗ ██╗     ███████╗██╗████████╗
██║     ██╔═══██╗██╔════╝██╔══██╗██║     ██╔════╝██║╚══██╔══╝
██║     ██║   ██║██║     ███████║██║     █████╗  ██║   ██║
██║     ██║   ██║██║     ██╔══██║██║     ██╔══╝  ██║   ██║
███████╗╚██████╔╝╚██████╗██║  ██║███████╗██║     ██║   ██║
╚══════╝ ╚═════╝  ╚═════╝╚═╝  ╚═╝╚══════╝╚═╝     ╚═╝   ╚═╝

Will it fit? Say what model you want — localfit figures out the rest.

Text and image generation. Fits locally? Run it. Doesn't fit? Free Kaggle GPU. Still too big? RunPod cloud. One command does everything: downloads models, starts servers, configures tools, launches your UI.

pip install localfit

One Command — Everything Works

# Chat + image gen in Open WebUI (gemma4 LLM + Flux Klein 4B images)
localfit launch openwebui --model gemma4:e4b --img klein-4b
# Code with image gen in localcoder
localfit launch localcoder --model gemma4:e4b --img klein-4b

# Claude Code with image MCP tools
localfit launch claude --model gemma4:e4b --img klein-4b

# Model doesn't fit locally? Run on free Kaggle GPU
localfit launch openwebui

…

View on GitHub

The full version

The complete write-up has the agent tool-calling results, the MCP setup where I handed the model control of a television, and the trick that gets llama.cpp decoding video when the server insists it cannot.

Read it on kartit.net

If you have hit this, I want your ioreg number and your chip. Drop them in the comments.

I Tested Every Gemma 4 Model Locally on My MacBook - What Actually Works

akartit — Sat, 04 Apr 2026 10:06:32 +0000

Audio ASR in 3 languages, image understanding, full-stack app generation, coding, and agentic behavior -- all running on a MacBook M4 Pro with 24GB RAM.

Interactive version with playable audio, live charts, and the working React app: gemma4-benchmark.pages.dev

Google just released Gemma 4 -- their new family of open-source multimodal models. Four sizes, Apache-2.0 licensed, supports text + image + audio.

I spent a day testing every variant. Real audio files. Real images. Code that has to compile and run. Here is my honest report.

The Gemma 4 Family

E2B -- Dense 2.3B, Text/Image/Audio, 4 GB at 4-bit. Phones and edge.
E4B -- Dense 4.5B, Text/Image/Audio, 5.5 GB at 4-bit. Laptops.
26B-A4B -- MoE 4B active/26B total, Text/Image, 16-18 GB at 4-bit.
31B -- Dense 31B, Text/Image, 17-20 GB at 4-bit. Maximum quality.

Speed Benchmarks

Ollama: E2B 95 tok/s | E4B 57 tok/s | 26B ~2 tok/s (swap) | 31B won't fit

Unsloth MLX: E2B 81 tok/s (3.6 GB) | E4B 49 tok/s (5.6 GB)

Ollama is 15-20% faster. Unsloth MLX uses 40% less memory.

Audio ASR: 3 Languages

Tested via Ollama OpenAI-compatible endpoint. Only E2B and E4B support audio.

Listen to all test audio samples: Audio Player

English ASR

E4B (1.0s): Perfect transcription. Every word correct with punctuation.

E2B (2.8s): Garbled -- missing words, no punctuation.

French ASR

E4B (1.6s): Perfect transcription with all French accents correct.

E2B (4.1s): Fragmented, missing most of the sentence.

Arabic ASR

E4B (6.0s): Perfect Arabic transcription -- every word correct.

E2B (6.0s): Garbled -- wrong words, disordered.

Speech Translation (E4B)

French to English: "All human beings are born free and equal in dignity and rights. They are endowed with reason and conscience..."

Arabic to English: "Hello, I am an artificial intelligence model. Today we will test speech recognition in the Arabic language..."

E4B is dramatically better than E2B for audio across all 3 languages.

Image Understanding

Test 1: Thai Temple -- Landmark Identification

E4B (54 tok/s): Thailand, Bangkok, Wat Phra Kaew (Temple of the Emerald Buddha) within the Grand Palace.

E2B (88 tok/s): Thailand, Bangkok, Grand Palace (less specific).

Test 2: AI-Generated Tokyo + Japanese OCR

AI-generated with nano-banana / Gemini

Both models correctly read Japanese kanji: 新宿ラーメン通り (Shinjuku Ramen Street)

Test 3: Venice Seagull

E4B: "A magnificent seagull perches watchfully atop a sculpted pedestal. The backdrop is a rich study in contrasting architectural styles..."

Full-Stack App Generation

E4B generated a 155-line working React + Tailwind Task Manager:

Try it live: gemma4-benchmark.pages.dev/task_manager.html

E2B failed -- code fragments instead of single file.

Coding: Compile and Run

Script	E2B	E4B
Fibonacci	PASS	PASS
Sieve of Eratosthenes	PASS	PASS
JSON processor	PASS	PASS
HTTP request	PASS	PASS
React single file	FAIL	PASS

Agentic Multi-Step Reasoning

6-step blog platform design. Both completed 6/6 steps. E4B output was 57% longer with more detail.

Why 26B Fails on 24GB

Community reports from r/LocalLLaMA suggest Gemma 4 has a KV cache memory issue (not verified on our hardware):

31B at 262K context: ~22GB just for KV cache (on top of model)
Google did not adopt KV-reducing techniques from Qwen 3.5
Workaround: --ctx-size 8192 --cache-type-k q4_0 --parallel 1

Official Benchmarks

Final Verdict

E4B -- The Sweet Spot -- 8.5/10

Perfect ASR in 3 languages. Working React app. Japanese OCR. 57 tok/s. 5.6 GB.

E2B -- Speed Demon -- 7/10

95 tok/s. 3.6 GB. Python works. Audio garbled. Failed complex HTML gen.

26B-A4B -- Heartbreaker -- 2/10 on 24GB

Amazing benchmarks (88.3% AIME). ~2 tok/s on 24GB. Needs 32GB+.

Quick Start

brew install ollama
ollama pull gemma4:e4b
ollama run gemma4:e4b

For 24GB MacBook: ollama run gemma4:e4b is the answer.

Tested April 3, 2026. MacBook Pro M4 Pro, 24GB, macOS Sequoia.

Interactive version: gemma4-benchmark.pages.dev

Sources: Google Model Card | HuggingFace Blog | Ollama | Unsloth Guide

How to Add Gemma 4 Models to OpenClaw (Fix Missing Model Error)

akartit — Fri, 03 Apr 2026 19:37:42 +0000

Google's Gemma 4 family just dropped — and it's a big deal. Open weights, Apache 2.0 license, multimodal, reasoning-capable, and free to use via the Gemini API. But if you're running OpenClaw as your AI assistant gateway, you'll hit a wall: OpenClaw doesn't have Gemma 4 in its built-in model catalog yet.

Here's how I fixed it in under 5 minutes — for both the 31B Dense and the 26B MoE (A4B) variants.

The Problem

After setting google/gemma-4-31b-it as my default model in OpenClaw, running openclaw models list showed:

Model                          Input   Ctx   Auth  Tags
google/gemma-4-31b-it          -       -     -     default,configured,missing

That missing tag means OpenClaw has the model name in config but no metadata — no input types, no context window size, no API protocol. It doesn't know how to talk to it.

Meanwhile, the raw API works fine:

#!/bin/bash
MODEL_ID="gemma-4-31b-it"
curl -X POST \
  -H "Content-Type: application/json" \
  "https://generativelanguage.googleapis.com/v1beta/models/${MODEL_ID}:streamGenerateContent?key=${GEMINI_API_KEY}" \
  -d '{
    "contents": [{
      "role": "user",
      "parts": [{"text": "Hello, what model are you?"}]
    }],
    "generationConfig": {
      "thinkingConfig": { "thinkingLevel": "HIGH" }
    },
    "tools": [{ "googleSearch": {} }]
  }'

Works perfectly. The problem is purely on OpenClaw's side.

The Fix: Register Custom Model Metadata

OpenClaw's config schema supports a top-level models block where you can inject model definitions that the built-in catalog doesn't have. You need to tell OpenClaw the API protocol, capabilities, and context window for each model.

Step 1: Edit your OpenClaw config

Open ~/.openclaw/openclaw.json and add a models block at the top level:

{
  "models": {
    "mode": "merge",
    "providers": {
      "google": {
        "baseUrl": "https://generativelanguage.googleapis.com/v1beta",
        "models": [
          {
            "id": "gemma-4-31b-it",
            "name": "Gemma 4 31B IT",
            "api": "google-generative-ai",
            "reasoning": true,
            "input": ["text", "image"],
            "contextWindow": 262144,
            "maxTokens": 131072
          },
          {
            "id": "gemma-4-26b-a4b-it",
            "name": "Gemma 4 26B A4B IT (MoE)",
            "api": "google-generative-ai",
            "reasoning": true,
            "input": ["text", "image"],
            "contextWindow": 262144,
            "maxTokens": 262144
          }
        ]
      }
    }
  }
}

What each field does

Field	Value	Why
`mode`	`"merge"`	Adds to the existing catalog instead of replacing it
`baseUrl`	Google's v1beta endpoint	Required by schema, even for the built-in Google provider
`api`	`"google-generative-ai"`	Tells OpenClaw to use Google's native API protocol (not OpenAI-compatible)
`reasoning`	`true`	Both models support configurable thinking modes
`input`	`["text", "image"]`	Multimodal: text + image (variable aspect ratio & resolution)
`contextWindow`	`262144`	262K context window (256K usable + overhead)
`maxTokens`	`131072` / `262144`	Max output tokens (31B: 131K, 26B MoE: 262K)

Step 2: Set up Google API auth

Make sure you have a Google auth profile in OpenClaw. If you don't:

openclaw models auth login
# Select "google" provider, "api_key" mode
# Paste your Gemini API key

Or set it via environment variable:

export GEMINI_API_KEY="your-key-here"

Step 3: Validate, restart, verify

# Validate config
openclaw config validate

# Restart the gateway
openclaw gateway restart

# Check models
openclaw models list

You should now see:

Model                          Input      Ctx    Auth  Tags
google/gemma-4-31b-it          text+image 256k   yes   default,configured
google/gemma-4-26b-a4b-it      text+image 256k   yes   configured

No more missing tag!

Step 4: Set your default and test

# Set default model
openclaw models set google/gemma-4-31b-it

# Test it
openclaw agent -m "Hello! What model are you?" --local --session-id test

Response:

"I am Gemma 4, a large language model developed by Google DeepMind."

Auto-Patch Script: One Command to Fix Everything

Don't want to edit JSON by hand? Save this script and run it — it patches your OpenClaw config automatically:

#!/bin/bash
# openclaw-gemma4-patch.sh
# Auto-patches OpenClaw config to add Gemma 4 models (31B Dense + 26B MoE)
# Usage: chmod +x openclaw-gemma4-patch.sh && ./openclaw-gemma4-patch.sh

set -euo pipefail

CONFIG="${OPENCLAW_CONFIG_PATH:-$HOME/.openclaw/openclaw.json}"
BACKUP="${CONFIG}.bak.$(date +%s)"

# --- Preflight checks ---
if ! command -v openclaw &>/dev/null; then
  echo "Error: openclaw not found in PATH" >&2
  exit 1
fi

if ! command -v python3 &>/dev/null; then
  echo "Error: python3 required for JSON patching" >&2
  exit 1
fi

if [ ! -f "$CONFIG" ]; then
  echo "Error: OpenClaw config not found at $CONFIG" >&2
  echo "Run 'openclaw configure' first or set OPENCLAW_CONFIG_PATH" >&2
  exit 1
fi

echo "Backing up config to $BACKUP"
cp "$CONFIG" "$BACKUP"

# --- Patch the config ---
python3 << 'PYEOF'
import json, sys, os

config_path = os.environ.get("OPENCLAW_CONFIG_PATH",
    os.path.expanduser("~/.openclaw/openclaw.json"))

with open(config_path, "r") as f:
    config = json.load(f)

gemma4_models = [
    {
        "id": "gemma-4-31b-it",
        "name": "Gemma 4 31B IT",
        "api": "google-generative-ai",
        "reasoning": True,
        "input": ["text", "image"],
        "contextWindow": 262144,
        "maxTokens": 131072
    },
    {
        "id": "gemma-4-26b-a4b-it",
        "name": "Gemma 4 26B A4B IT (MoE)",
        "api": "google-generative-ai",
        "reasoning": True,
        "input": ["text", "image"],
        "contextWindow": 262144,
        "maxTokens": 262144
    }
]

if "models" not in config:
    config["models"] = {"mode": "merge", "providers": {}}
if "providers" not in config["models"]:
    config["models"]["providers"] = {}
if "google" not in config["models"]["providers"]:
    config["models"]["providers"]["google"] = {
        "baseUrl": "https://generativelanguage.googleapis.com/v1beta",
        "models": []
    }

google = config["models"]["providers"]["google"]
if "baseUrl" not in google:
    google["baseUrl"] = "https://generativelanguage.googleapis.com/v1beta"
if "models" not in google:
    google["models"] = []

existing_ids = {m["id"] for m in google["models"]}
added = []
for model in gemma4_models:
    if model["id"] not in existing_ids:
        google["models"].append(model)
        added.append(model["id"])

if "agents" in config and "defaults" in config["agents"]:
    defaults = config["agents"]["defaults"]
    if "models" not in defaults:
        defaults["models"] = {}
    for model in gemma4_models:
        key = f"google/{model['id']}"
        if key not in defaults["models"]:
            defaults["models"][key] = {}

ordered = {"models": config.pop("models")}
ordered.update(config)

with open(config_path, "w") as f:
    json.dump(ordered, f, indent=2)
    f.write("\n")

if added:
    print(f"Added models: {', '.join(added)}")
else:
    print("Models already present, no changes needed")
PYEOF

# --- Validate ---
echo ""
echo "Validating config..."
if ! openclaw config validate; then
  echo "Validation failed! Restoring backup..."
  cp "$BACKUP" "$CONFIG"
  echo "Restored. Please check your config manually."
  exit 1
fi

# --- Restart gateway ---
echo ""
echo "Restarting gateway..."
openclaw gateway restart

# --- Verify ---
echo ""
echo "Verifying models..."
openclaw models list

echo ""
echo "Done! Gemma 4 models are ready."
echo "Set your default with: openclaw models set google/gemma-4-31b-it"
echo "Test with: openclaw agent -m 'Hello!' --local --session-id test-gemma"

Save it and run:

chmod +x openclaw-gemma4-patch.sh
./openclaw-gemma4-patch.sh

The script is idempotent — running it twice won't duplicate models. It backs up your config before patching and auto-rolls back if validation fails.

Testing Both Models

Once patched, verify both variants respond:

# Test the 31B Dense model
openclaw models set google/gemma-4-31b-it
openclaw agent -m "What model are you? One sentence." \
  --local --session-id test-31b --json | python3 -c "
import json,sys
r = json.load(sys.stdin)
print(f'Model: {r[\"meta\"][\"agentMeta\"][\"model\"]}')
print(f'Response: {r[\"payloads\"][0][\"text\"]}')
print(f'Time: {r[\"meta\"][\"durationMs\"]}ms')
"

# Test the 26B MoE model
openclaw models set google/gemma-4-26b-a4b-it
openclaw agent -m "What model are you? One sentence." \
  --local --session-id test-26b --json | python3 -c "
import json,sys
r = json.load(sys.stdin)
print(f'Model: {r[\"meta\"][\"agentMeta\"][\"model\"]}')
print(f'Response: {r[\"payloads\"][0][\"text\"]}')
print(f'Time: {r[\"meta\"][\"durationMs\"]}ms')
"

# Switch back to your preferred default
openclaw models set google/gemma-4-31b-it

Test via Raw curl (No OpenClaw)

Verify your API key works before patching:

#!/bin/bash
# test-gemma4-api.sh — Quick API smoke test for both Gemma 4 models
set -e -E

if [ -z "$GEMINI_API_KEY" ]; then
  echo "Error: Set GEMINI_API_KEY first" >&2
  exit 1
fi

for MODEL in gemma-4-31b-it gemma-4-26b-a4b-it; do
  echo "--- Testing $MODEL ---"
  curl -s -X POST \
    -H "Content-Type: application/json" \
    "https://generativelanguage.googleapis.com/v1beta/models/${MODEL}:generateContent?key=${GEMINI_API_KEY}" \
    -d '{
      "contents": [{
        "role": "user",
        "parts": [{"text": "What model are you? Reply in one sentence."}]
      }],
      "generationConfig": {
        "thinkingConfig": { "thinkingLevel": "HIGH" }
      }
    }' | python3 -c "
import json,sys
r = json.load(sys.stdin)
for part in r.get('candidates',[{}])[0].get('content',{}).get('parts',[]):
    if 'text' in part:
        print(part['text'])
        break
" 2>/dev/null || echo "FAILED"
  echo ""
done

How I Diagnosed This

openclaw models list — showed model as missing (no metadata)
openclaw config schema — extracted the full JSON schema to find the exact format for custom model definitions
Found the supported api protocols: google-generative-ai, openai-completions, anthropic-messages, ollama, and others
Added the config block, validated with openclaw config validate
Restarted gateway and confirmed with a test message

About Gemma 4

Gemma 4 is Google's latest open model family with three architectures:

Model	Type	Parameters	Active Params	Context	Memory (BF16)
Gemma 4 E2B	Efficient	2B effective	—	128K	9.6 GB
Gemma 4 E4B	Efficient	4B effective	—	128K	15 GB
Gemma 4 31B	Dense	31B	31B	256K	58.3 GB
Gemma 4 26B A4B	MoE	26B	4B	256K	48 GB

Key capabilities:

Reasoning with configurable thinking modes (OFF/LOW/MEDIUM/HIGH)
Multimodal: text, image (all models), video & audio (E2B/E4B)
Function calling for agentic workflows
Native system prompt support
Apache 2.0 license — fully open for commercial use

Free Tier Rate Limits (Gemini API)

Model	RPM (Requests/min)	TPM (Tokens/min)
Gemma 4 26B	3 / 15	77 / Unlimited
Gemma 4 31B	3 / 15	15.04K / Unlimited

The 31B gets significantly more token throughput on free tier. Both work for experimentation and personal assistant use cases.

The General Pattern: Adding Any Unsupported Model

This same approach works for any model from any provider that OpenClaw doesn't ship in its catalog:

Verify the API key works outside OpenClaw (use curl)
Find the right api protocol from openclaw config schema — options include:
- google-generative-ai
- openai-completions
- openai-responses
- anthropic-messages
- ollama
- bedrock-converse-stream
- github-copilot
- azure-openai-responses
Add the models.providers.<name> block with id, api, contextWindow, maxTokens
Validate and restart: openclaw config validate && openclaw gateway restart

That's it. Sixty seconds from "unknown model" to a working agent.

Links: