DEV Community: Tejas Patil

Gemma 4's Multi-Token Prediction Changes the Economics of Running AI Locally — Here's the Full Breakdown

Tejas Patil — Sun, 24 May 2026 06:34:05 +0000

This is a submission for the Gemma 4 Challenge: Write About Gemma 4

There's a hard wall that every developer hits when they try to run a capable AI model locally. It's not the GPU. It's not the RAM. It's the memory bandwidth.

Standard autoregressive generation — the way every LLM has worked since GPT-2 — does one thing at a time: predict a token, move that token back through the model, predict the next one. Each step requires shipping gigabytes of weight matrices from memory to the processor. On a MacBook, an RTX 4080, or a cloud instance you're paying $0.40/hour for, this shuffle is the bottleneck. More VRAM doesn't fix it. Faster GPUs barely dent it. It's a structural constraint baked into how transformers generate text.

On May 5, 2026, Google shipped the fix. Multi-Token Prediction (MTP) drafters for the entire Gemma 4 family — and the numbers are real: up to 3x faster inference, zero quality loss, Apache 2.0 licensed, works with Ollama, vLLM, Hugging Face, MLX, SGLang, and LiteRT-LM out of the box.

This is the most important thing that happened to local AI this month. Let me show you exactly why — and help you figure out which Gemma 4 model is actually right for your use case.

First: The Four Models Explained

Gemma 4 isn't one model. It's a family of four, each designed for a genuinely different deployment context. Getting the model selection right matters as much as understanding MTP.

E2B — The Pocket Rocket

"E" stands for "effective" parameters. The E2B weighs in at roughly 1.5GB at 4-bit, runs on modern Android phones via Google AICore, works completely offline, and natively understands audio and images. It has a 128K context window.

The trick behind its size efficiency is Per-Layer Embeddings (PLE) — instead of stacking more transformer layers, each decoder layer gets its own small embedding table per token. The static weight footprint is technically larger than 2B parameters might suggest, but the active compute stays tiny. The result: a model that can live on a Raspberry Pi or a mid-range Android phone and still reason across an entire book chapter in one shot.

Use it when: you're building a mobile app, an offline tool, an IoT integration, or anything that must run without a network connection.

E4B — The Edge Sweet Spot

Same architecture philosophy as E2B, more headroom. Runs in ~5GB RAM at 4-bit, ~15GB at full 16-bit. Also supports audio and image input natively. Also 128K context.

The E4B hits the crossover point where capability meets practicality for most developer laptops. You're not giving up much compared to the bigger models for typical tasks — coding assistance, document Q&A, image analysis — and you keep the low-latency edge advantage.

Use it when: you're building a local-first desktop app, a developer tool, or anything running on a laptop that needs genuine multimodal capability.

26B A4B — The Efficiency Cheat Code

This is the sneaky one. 26 billion total parameters, but only 4 billion activate during any given inference. It's a Mixture-of-Experts (MoE) architecture: the model routes each token through the 4B expert subset most relevant to that input, ignoring the rest. All 26B must be loaded into memory (~18GB at 4-bit), but the compute per token is closer to a 4B model.

The result: #6 open model in the world on the Arena AI leaderboard, outcompeting models 20x its size, running at 4B-like speeds, with a 256K context window.

Use it when: you have ~20GB VRAM (RTX 3090, 4090, A10G) and want near-frontier capability with fast inference. This is the production sweet spot for most self-hosted deployments.

31B Dense — The Flagship

The 31B is currently #3 open model in the world on the Arena AI text leaderboard. Dense architecture (no MoE routing), 256K context, 20GB at 4-bit or 34GB at 8-bit. The most capable in the family, the most hardware-hungry.

Use it when: you need maximum quality and have the iron to back it up — A100, H100, multi-GPU setups, or high-memory cloud instances.

Deep Dive: How MTP Actually Works

Now for the part that changes everything.

The core insight behind Multi-Token Prediction is that the big, slow target model doesn't need to do all the work. A small, fast drafter model can predict several tokens ahead speculatively — and the target model can verify all of them in parallel in a single forward pass.

Here's the pipeline step by step:

Step 1 — Draft. The drafter (a compact model purpose-built for this) takes the current sequence and rapidly predicts 4–8 tokens ahead. This is cheap: the drafter is small, and it runs quickly.

Step 2 — Verify. The full target model (E2B, 26B A4B, whatever you're using) processes all the drafted tokens simultaneously in one forward pass. It checks each one.

Step 3 — Accept or Correct. If the target model agrees with a drafted token, it's accepted for free. If it disagrees, it generates the correct token for that position and the drafter starts fresh from there. Importantly, even a rejected step isn't wasted — the target model always produces the correct token at that position.

Net result: The target model does dramatically fewer forward passes per output token. The memory bandwidth bottleneck still exists, but you hit it far less often. Hence the 3x speedup.

What Makes Gemma 4's MTP Different

Here's the part that genuinely separates this from what others are doing.

KV cache sharing. The Key-Value cache (the model's short-term memory for attention values) is shared between the drafter and the target model. On a memory-constrained device, this is critical — no duplicating data in VRAM, no cache invalidation overhead.

Shared target activations. The drafter doesn't start from scratch. It uses the internal representations — the "activations" — that the target model has already computed in its deeper layers. The drafter is piggybacking on work already done. This makes the draft step faster and more accurate.

Official, first-party, Apache 2.0. Llama, Qwen, and DeepSeek all train MTP-aware variants. None of them ship official drafter checkpoints. Community drafters exist for those models, but the quality is uneven and the integration is manual. Gemma 4 ships polished, purpose-built drafters as standalone checkpoints on Hugging Face and Kaggle, with runtime support already baked into Ollama, vLLM, SGLang, MLX, and Hugging Face Transformers. It's one config flag, not a research project.

The Hardware Math (This Is Where It Gets Interesting)

The 26B A4B model, with MTP enabled, running on a cloud instance with 20GB VRAM:

Instance cost: ~$0.40–$0.80/hour (RTX A10G class)
MTP throughput improvement: ~2.5–3x over baseline
Per-token cost at sustained inference: competitive with GPT-4o mini pricing

That's the sentence that changes the build-vs-hosted calculus for a lot of teams. "Competitive with GPT-4o mini" at a capability level that places the model in the top 10 open models globally, on hardware you fully control, with data that never leaves your infrastructure, under a license with no MAU limits and no royalty clauses.

For mobile: the E2B with MTP runs on Android via Google AI Edge Gallery. The efficient embedder in the E-series models further reduces the compute overhead of the drafter on constrained hardware. A 3x speedup on a phone means the difference between a model that feels native and one that feels like it's thinking.

Setting It Up (This Takes About 10 Minutes)

With Ollama:

# Pull the 26B MoE model + its MTP drafter
ollama pull gemma4:26b
ollama pull gemma4:26b-mtp-drafter

# Run with speculative decoding enabled
ollama run gemma4:26b --speculative-model gemma4:26b-mtp-drafter

With vLLM:

from vllm import LLM, SamplingParams

llm = LLM(
    model="google/gemma-4-26B-A4B-it",
    speculative_model="google/gemma-4-26B-A4B-mtp-drafter",
    num_speculative_tokens=5,
    tensor_parallel_size=1
)

sampling_params = SamplingParams(temperature=0.7, max_tokens=512)
outputs = llm.generate(["Explain speculative decoding in plain English."], sampling_params)
print(outputs[0].outputs[0].text)

With Hugging Face Transformers:

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

tokenizer = AutoTokenizer.from_pretrained("google/gemma-4-26B-A4B-it")
target = AutoModelForCausalLM.from_pretrained(
    "google/gemma-4-26B-A4B-it", 
    torch_dtype=torch.bfloat16,
    device_map="auto"
)
drafter = AutoModelForCausalLM.from_pretrained(
    "google/gemma-4-26B-A4B-mtp-drafter",
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

inputs = tokenizer("Walk me through the 128K context window use case:", return_tensors="pt").to("cuda")

outputs = target.generate(
    **inputs,
    assistant_model=drafter,
    max_new_tokens=300
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Which Model Should You Actually Pick?

Here's the decision tree I'd give a colleague:

Are you building for mobile or IoT?
→ E2B. No competition. 1.5GB, offline, audio-native, Apache 2.0.

Are you building a local-first desktop tool or developer assistant?
→ E4B with MTP drafter. Best balance of speed and capability for a laptop GPU.

Are you self-hosting for a production SaaS or internal tool?
→ 26B A4B with MTP drafter. MoE gives you near-31B quality at 4B inference speed. The economics work at scale.

Do you need absolute maximum quality and have A100/H100 infrastructure?
→ 31B Dense with MTP drafter. #3 open model in the world. That's the ceiling of what you can run yourself right now.

The Bigger Picture

Here's my honest take after spending time with Gemma 4 and its MTP release: we just crossed a threshold.

The 31B model ranking #3 globally among open models is remarkable. But it would be table stakes — every major lab has a flagship. What makes Gemma 4 significant is the combination: frontier-level capability at the top, a model that runs in 1.5GB on a phone at the bottom, and MTP drafters that make all of them dramatically faster, all under a license with no strings attached.

The MTP implementation specifically matters because it signals something about Google's intent. This isn't a capability demo — it's infrastructure. Shipping official, polished, first-party drafter checkpoints that plug into every major serving framework in a single afternoon is the kind of work that benefits the entire open-weight ecosystem, not just Gemma users.

The other labs will follow. Llama and Qwen will ship official drafters. The bar just moved.

For developers: the "should I use an API or run it myself" question just got a lot more interesting. For the first time, the answer for a lot of production workloads might genuinely be "run it yourself, it's cheaper, it's faster, and you own the data."

That is a real change. And Gemma 4 MTP is the specific reason it's true now when it wasn't true six months ago.

Resources

Official MTP Overview — Google AI for Developers
MTP with Hugging Face Transformers — full implementation guide
Gemma 4 on Ollama — one-command local setup
Gemma 4 on Hugging Face — all model checkpoints
Google AI Edge Gallery — try E2B/E4B MTP on Android or iOS

What are you building with Gemma 4? I'm particularly curious who's running the E2B on actual edge hardware — drop it in the comments.

WebMCP Is the Most Important Thing Google Announced at I/O 2026 (And Almost Nobody Is Talking About It)

Tejas Patil — Sat, 23 May 2026 22:19:01 +0000

This is a submission for the Google I/O Writing Challenge

Right now, every AI agent that tries to use a website is basically doing this:

Take a screenshot
Guess what's on screen
Click something and hope
Take another screenshot
Repeat until it works or gives up

It's the digital equivalent of reading someone's lips through a frosted glass window. It kind of works. It's slow, expensive, and breaks constantly on anything slightly dynamic — a modal, a lazy-loaded form, a JS-rendered button.

Google's answer to this is called WebMCP — Web Model Context Protocol. It entered a public origin trial in Chrome 149 on May 19, 2026, during the I/O Developer keynote. And I think it's the most consequential announcement of the whole event — not because of what it does today, but because of what it signals about where the web is going.

Let me show you what it actually is, how to use it right now, and why I have real questions about whether it will succeed.

What WebMCP Actually Does

The idea is simple: instead of making AI agents figure out what your website does by staring at it, you tell them explicitly.

WebMCP lets you expose structured tools — JavaScript functions and annotated HTML forms — directly to browser-based AI agents. The agent doesn't scrape. It calls your tool like an API.

There are two ways to implement it:

The Declarative API (for forms)

You annotate existing HTML forms with a data-mcp-tool attribute and a description. The agent reads the annotation and knows exactly what the form does.







All categories
Electronics
Clothing


  Search

That's it. An agent seeing this form no longer has to guess what the fields mean or what the form does. You've told it.

The Imperative API (for JavaScript functions)

For more complex interactions, you register tools programmatically:

navigator.mcp.registerTool({
  name: "add_to_cart",
  description: "Add a product to the shopping cart by product ID and quantity",
  parameters: {
    productId: {
      type: "string",
      description: "The unique product identifier"
    },
    quantity: {
      type: "number",
      description: "Number of units to add",
      minimum: 1
    }
  },
  handler: async ({ productId, quantity }) => {
    const result = await cartService.add(productId, quantity);
    return { success: true, cartTotal: result.total };
  }
});

An agent calling add_to_cart with { productId: "ABC123", quantity: 2 } will get a reliable result — no screenshot guessing, no DOM parsing, no retries.

Why I'm Genuinely Excited

1. This is a Google + Microsoft co-project

This is the detail that changes everything for adoption: WebMCP is developed jointly by Google and Microsoft in the W3C Web Machine Learning Community Group.

That's not just a Google standard. It's an emerging web standard with two of the biggest browser vendors aligned on the spec from day one. Cross-vendor agreement at this stage is rare and meaningful. It substantially increases the chance this becomes a real, lasting part of the web platform.

2. The timing is right

Browser agents — AI systems that navigate websites on your behalf — are growing fast. Gemini in Chrome, which will support WebMCP APIs, is one. Others are coming. Right now these agents are all fighting the same brittle DOM-scraping battle. WebMCP gives the web a way to meet them halfway.

Implementing WebMCP on your site today is the same category of investment as adding proper aria-label attributes in 2015 or adding og:title meta tags in 2012. It felt optional then. It became table stakes.

3. The developer experience is genuinely low-friction

The declarative API requires zero new JavaScript — just HTML annotations. You can expose your most common user flows to agents in an afternoon. The barrier is low enough that "let's try it" is a reasonable thing to say at a sprint planning meeting right now.

Where I Have Real Questions

I don't want to just be a hype machine, because there are genuine open questions here.

Firefox and Safari haven't committed

This is the elephant in the room. Mozilla and Apple have not signed on to WebMCP. For a standard to truly succeed on the web, it needs more than Chrome. Right now, if you implement WebMCP, it's Chrome-only by design.

That's not fatal — lots of meaningful features started as Chrome-only experiments before getting broader adoption. But it's a real constraint. If your user base is heavy on Safari (mobile web, Apple users), WebMCP tooling won't work for those agents browsing on Safari.

"No headless support" is a meaningful limitation

The official Chrome documentation is explicit: WebMCP requires a browser tab to be open. There's no support for agents to call your tools in a headless state.

This means WebMCP is specifically for in-browser agent interactions — not for server-side automation pipelines that many enterprise workflows rely on. For those use cases, you'd still need a backend MCP server. WebMCP and server-side MCP are complementary, not interchangeable.

The spec is not yet on the W3C official standards track

It currently lives in the W3C Web Machine Learning Community Group — an incubation space, not the full standards process. The path from origin trial to official web standard is long and uncertain. WebMCP could follow the path of Service Workers (proposed → standard → ubiquitous). Or it could follow the path of a dozen other promising origin trials that never made it.

What I'd Actually Recommend

If you maintain a web app with forms or user-facing workflows, here's what I'd do this week:

Step 1: Enable the flag in Chrome today
Go to chrome://flags and search for "WebMCP". Set it to Enabled, relaunch, and you can start testing immediately without waiting for Chrome 149.

Step 2: Pick your one most important user flow
Don't try to annotate everything. Pick the single form or interaction that an agent would most benefit from — a search form, a checkout step, a filter UI. Annotate it with the declarative API. It'll take an hour.

Step 3: Sign up for the origin trial
Visit the Chrome origin trial page and register your domain for the WebMCP trial. This lets you ship WebMCP support to real users before Chrome 149 hits stable.

Step 4: Watch what happens when Gemini in Chrome supports it
This is the moment that will make the investment pay off. When Google's in-browser agent can call your registered tools directly — that's when the "I annotated my forms" work starts delivering real value.

The Bigger Picture

Here's my actual take after sitting with I/O 2026 for a few days:

The Gemini model announcements are table stakes at this point. Every major AI lab releases faster, cheaper models every few months. That's not a story; it's a cadence.

WebMCP is different. It's infrastructure. It's Google (and Microsoft) trying to answer a structural question about the web's future: when AI agents become first-class citizens of the browser, what contract does a website make with them?

The answer they're proposing is WebMCP: an explicit, structured, queryable tool surface that gives agents what they actually need instead of forcing them to infer it.

If that standard gets adopted, it changes how we think about building for the web. We'll think about our web apps as having three user types: humans on desktop, humans on mobile, and AI agents. WebMCP is the API layer for the third type.

That is a genuinely new idea. And it came from a developer keynote that most people stopped watching after the Gemini 3.5 Flash benchmarks.

Are you going to try WebMCP in the origin trial? I'd love to hear which use cases you're thinking about — drop them in the comments.