This is a submission for the Gemma 4 Challenge: Write About Gemma 4
I want to tell you about the moment I stopped treating local AI as a compromise.
I was building a UX analysis tool. The idea was simple: give it a screenshot, get back a structured critique grounded in Nielsen's heuristics, cognitive load theory, WCAG — the real frameworks, not vibes. I'd used cloud models before. They work. But they also mean your users' screenshots leave their machine, hit an API, and cost money per call.
So I pulled Gemma 4 E4B locally via Ollama, pointed it at Stripe.com, and waited.
Fifty-six seconds later it handed me a structured JSON object with an overall UX score, three specific friction points each mapped to a Nielsen heuristic, accessibility flags, layout analysis, and prioritized recommendations with effort and impact ratings.
On CPU. No GPU. No API key. No data leaving the machine.
That's when the compromise framing broke for me. This isn't a fallback. It's a capability shift.
What Gemma 4 Actually Is
Gemma 4 is Google's latest open-weights model family, and the headline feature is multimodality — it can see images, not just read text. But the more interesting story is the range of what Google shipped:
| Variant | Parameters | Context Window | Best For |
|---|---|---|---|
| E2B | ~2B | 8K | Edge, mobile, fast inference |
| E4B | ~4B | 8K | Local development, CPU-viable |
| 31B Dense | ~31B | 128K | Complex reasoning, long documents |
Three very different tools with the same name. Picking wrong costs you either quality or speed. Let me tell you what I've actually learned about each.
E2B: The One That Fits Anywhere
The 2-billion-parameter model is built for constraints. Think browser extensions, mobile apps, Raspberry Pi, anything where you cannot afford to wait or cannot carry a heavy runtime.
It handles basic instruction-following and short-form generation well. What it struggles with is structured output — complex JSON schemas, multi-step reasoning chains, tasks that require holding a lot of context simultaneously. If your task is "summarize this paragraph" or "classify this text into one of five categories," E2B is excellent and blazingly fast. If your task requires the model to reason across multiple frameworks while producing a nested JSON object, you'll feel the ceiling.
Use E2B when: speed > depth, you're running on constrained hardware, or the task is classifying/summarizing rather than reasoning.
E4B: The Sweet Spot for Developers
This is the one I use. The 4-billion-parameter model is the first point in the family where you stop noticing the limitations on most practical tasks.
What changed for me: E4B can follow a complex, multi-section JSON schema reliably. I give it a system prompt that defines eight nested fields — cognitiveLoad, trustScore, frictionPoints, recommendations, accessibilityFlags, layoutAnalysis — and it fills them correctly, with real observations, not hallucinated filler.
It also handles format: "json" mode in Ollama properly. This flag constrains the output to valid JSON, and E4B respects it while still producing coherent, specific content. E2B can struggle here — the output is valid JSON but the reasoning quality inside the strings drops noticeably.
On a modern CPU (M-series Mac or reasonably recent x86), E4B runs in under two minutes for most tasks. That's usable. It's not instant, but it's not "go make a coffee" either. On a GPU it's much faster.
Use E4B when: you need structured output, multimodal input, or reasoning quality that holds up in a real product — and you want it to run locally without a GPU.
31B Dense: When You Need to Think Hard
The 31B model is a different kind of tool. The 128K context window alone opens up use cases the smaller models simply cannot touch: full codebase analysis, long-document reasoning, multi-turn conversations that span thousands of tokens without losing thread.
The reasoning depth is also qualitatively different. Tasks that require synthesizing across many pieces of information — legal document review, detailed technical explanations, nuanced comparative analysis — land differently at 31B than at 4B. The model doesn't just answer; it considers.
The tradeoff is infrastructure. You need a GPU (or significant RAM for CPU inference), and the latency is real. This isn't a laptop model. But if you're building something that needs serious reasoning and you can provision the compute, 31B Dense is competitive with much larger proprietary models on a lot of benchmarks.
Use 31B Dense when: you have the compute, your task requires deep reasoning or long context, and quality is the primary constraint.
The Feature That Changed How I Build: Multimodal + JSON Mode
Here's the combination that makes Gemma 4 genuinely special for application developers.
Most LLM applications are text in, text out. You send a prompt, you parse the response, you deal with whatever format the model decided to use that day. Gemma 4 changes two things at once:
1. You can send images. Not OCR'd text from images. Not bounding boxes. The actual image. The model reasons about what it sees — colors, layout, spatial relationships, visual hierarchy.
2. You can enforce JSON output. With format: "json" in the Ollama request, the model outputs valid JSON. Combined with a well-defined system prompt that describes the exact schema you want, you get structured, machine-readable output you can render directly in a UI.
Here's what that looks like in practice:
const response = await fetch("http://localhost:11434/api/generate", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
model: "gemma4:e4b",
prompt: systemPrompt + "\n\n" + userPrompt,
images: [base64Image], // raw base64, no data URI prefix
format: "json", // constrain output to valid JSON
stream: true,
options: {
temperature: 0.3, // lower = more deterministic structure
num_ctx: 8192,
},
}),
});
The model receives the image, reasons about it, and returns a structured JSON object. No parsing heuristics. No regex. No "hope the model uses the format I asked for." Validate it with Zod or any schema library and you're done.
One practical note: sometimes the first pass at temperature: 0.3 produces a malformed response, especially on complex schemas. A simple retry at temperature: 0.1 with streaming disabled almost always fixes it:
// Retry pattern for malformed JSON
const retryResponse = await fetch("http://localhost:11434/api/generate", {
method: "POST",
body: JSON.stringify({
model: "gemma4:e4b",
prompt: systemPrompt + "\n\n" + userPrompt,
images: [base64Image],
format: "json",
stream: false, // non-streaming for retry
options: { temperature: 0.1, num_ctx: 8192 },
}),
});
This two-pass pattern makes structured multimodal output production-reliable.
Getting Started in Five Minutes
If you want to run Gemma 4 locally right now:
# Install Ollama (macOS/Linux)
curl -fsSL https://ollama.com/install.sh | sh
# Pull the E4B model (~3GB download)
ollama pull gemma4:e4b
# Test it immediately
ollama run gemma4:e4b "Explain the difference between cognitive load and visual noise in UX design"
For multimodal input via API:
# Send an image (base64 encoded)
curl http://localhost:11434/api/generate -d '{
"model": "gemma4:e4b",
"prompt": "Describe the visual hierarchy in this UI screenshot",
"images": ["'$(base64 -i your-screenshot.png)'"],
"format": "json",
"stream": false
}'
That's it. No API key, no rate limits, no billing dashboard.
What This Actually Means
I think we're underreacting to what's happening here.
A 4-billion-parameter multimodal model that runs on a consumer laptop, follows complex structured output schemas, reasons about images, and produces output good enough to use in a real product — that's not a prototype capability. That's a production capability.
For developers, this changes the build calculus for a whole category of applications. Anything that involves user data you'd rather not send to a cloud API. Any tool that needs to work offline. Any product targeting users who are rightfully skeptical about where their data goes. Any hobby project where you don't want to watch a usage bill grow.
For the broader ecosystem, a capable open-weights multimodal model means researchers, educators, and developers in places with expensive or unreliable internet access can run serious AI locally. The capability is no longer gated behind an API key and a credit card.
The question I keep coming back to: if a 4B model running on CPU can do this now, what does the roadmap look like in two years? We're probably not near the ceiling.
I'm not sure what that means yet. But I'm fairly sure the "local AI as compromise" framing is already wrong, and most people haven't noticed.
Running Gemma 4 E4B locally via Ollama. Models are open-weights and available at ollama.com/library/gemma4.
Top comments (0)