Measuring Gemma & On-Device Latency on 16GB Apple Silicon for Agent Workflows

Before you wire up a multi-agent workflow, measure how fast your local models actually respond.

Model leaderboards rank quality, and spec sheets list parameter counts. Neither tells you how long a user, or the next agent in your chain, waits before the first token shows up. In an agent workflow, a router or triage step runs on every request. A few hundred milliseconds of extra latency there gets paid on every call.

So I measured time-to-first-token (TTFT) and end-to-end latency (E2E) for several small local models on a modest machine, before choosing any of them.

Test setup

• Hardware: M1 MacBook Air, 16GB unified memory, running on battery
• OS: macOS 27.2
• Sampling: temperature=0, 512 max output tokens
• Method: the median of 5 formal runs per model

Models tested, all running locally:

Model	Runtime	Notes
Apple fm system	Local HTTP via fm serve	Apple's on-device system model
Google Gemma 4 E2B	MLX, 4-bit	MLX community revision 23876752
Google Gemma 4 E4B	MLX, 4-bit	MLX community revision 475b9088
Microsoft Phi-4 Mini	MLX, 4-bit	MLX community revision ac1c269c
Mistral Ministral 3 3B	MLX, 4-bit	MLX community revision a962dcb0

The results

Medians of 5 formal runs each:

Model	TTFT	E2E	Prompt tokens	Gen tokens	Peak RAM
Apple fm system	0.693s	0.886s	n/a	n/a	n/a
Gemma 4 E2B (4-bit)	0.152s	1.075s	61	34	2.76 GB
Phi-4 Mini (4-bit)	0.358s	1.670s	58	32	2.37 GB
Ministral 3 3B (empty system + regex fixed)	0.564s	1.764s	72	34	2.14 GB
Gemma 4 E4B (4-bit)	0.398s	2.313s	61	37	4.38 GB
Ministral 3 3B (default template, long system message)	3.231s	4.496s	602	33	n/a

Two models lead, on different metrics:

• Gemma 4 E2B has the fastest TTFT at 152ms. The first token arrives almost immediately.
• Apple fm system has the fastest E2E at 886ms for these short, single-sentence outputs, even though its TTFT (0.693s) is much slower than Gemma E2B's.

Which model is fastest depends on which latency you care about.

TTFT and E2E answer different questions

TTFT is how long you wait before anything happens. It governs perceived responsiveness for interactive UIs and streaming. It also governs how quickly a classifier or router can start returning its decision.

E2E is the total time until the full response is complete. For short outputs, it tells you how long a blocking call takes.

Gemma E2B's profile is a very fast start with a slightly longer total (0.152s TTFT, 1.075s E2E). Apple FM starts slower but finishes the short answer sooner (0.693s TTFT, 0.886s E2E). If your agent needs to react the moment a request lands, TTFT matters most. If it is making blocking calls that return a short sentence, E2E matters more.

The two metrics are not interchangeable, so a benchmark that reports only one can mislead you.

The prompt template trap

The most useful finding was not about any model. It was about the prompt.

Ministral 3 3B appears twice in the table:

• With an empty system message and a fixed regex: 72 prompt tokens, 0.564s TTFT, 1.764s E2E
• With its default template and long system message: 602 prompt tokens, 3.231s TTFT, 4.496s E2E

The generated output was about the same size in both runs (34 vs 33 tokens). The model and hardware did not change. Only the template changed, and it added about 530 prompt tokens that had to be processed before the first token. TTFT went from 0.56s to 3.23s.

For fast agent routing, system prompt bloat is the real killer. Before you blame the model or the hardware, check how many tokens your default chat template injects. If you are running a short classification or tool-routing step, a lean prompt can matter more than the choice of model.

Non-thinking vs thinking modes

I also verified the token overhead at the template level for non-thinking and thinking modes. The reasoning is the same: any extra tokens the template adds ahead of your actual prompt show up as latency. Check the rendered prompt your runtime sends, not just the prompt you wrote.

Stop reason is not accuracy

A run that finishes with a clean stop reason has only told you that generation ended. It has not told you the answer was right.

When I read the actual outputs, I found edge cases that latency numbers would never show:

• Date boundary drift: dates near boundaries came back shifted.
• Missing owners: extracted action items sometimes dropped the person responsible.

A fast, wrong answer is still wrong. Treat latency as a filter for choosing candidate models, then check each candidate's outputs against your task before you commit to it.

Memory footprint on 16GB

The peak RAM numbers for the models that reported it:

• Ministral 3 3B: 2.14 GB
• Phi-4 Mini: 2.37 GB
• Gemma 4 E2B: 2.76 GB
• Gemma 4 E4B: 4.38 GB

On a 16GB machine, all of these fit. But your agent runtime, editor, browser, and other tools share the same unified memory. A small model that stays resident while you work is easier to live with than one that competes for memory. Gemma E4B costs roughly 1.6 GB more than E2B, and in this test it was slower on both TTFT (0.398s vs 0.152s) and E2E (2.313s vs 1.075s).

What this means for agent workflows

A common pattern for agent systems is to split the work:

1. A small local model handles the fast, frequent decisions: intent classification, triage, tool routing, simple extraction.
2. A larger cloud model handles heavy reasoning: planning, long-context analysis, and complex multi-step work, for example Gemini 2.5 Flash / Pro or Google Antigravity.

The measurements support this split. Gemma 4 E2B's 152ms TTFT is well suited to the first step. Routing happens on every request, so a sub-second start with a modest 2.76 GB footprint adds little overhead. Apple's fm system is a reasonable fit for quick one-sentence outputs, given its 886ms E2E. Neither is a replacement for a frontier model on hard reasoning, and I did not test them for it.

Architecting with Gemma & Gemini

For Google developers building agentic workflows, pairing on-device Gemma 4 E2B as a fast, zero-network dispatcher with cloud Gemini 2.5 Flash / Pro for complex reasoning gives the optimal balance between perceived latency and reasoning depth:

• Local Dispatcher (Gemma 4 E2B): Delivers instantaneous ~150ms TTFT to classify intents or parse local CLI parameters without incurring cloud round-trip latency (~300ms+) or API quota.
• Cloud Reasoner (Gemini): Takes over when tasks require deep context understanding, multi-turn tool calling, or complex synthesis.

Takeaways

1. Measure TTFT and E2E separately. They rank the models differently, and they matter for different parts of a workflow.
2. Audit your prompt template. A default system message took one model from 72 to 602 prompt tokens and its TTFT from 0.56s to 3.23s.
3. Check outputs, not just stop reasons. Date drift and missing owners got past a clean finish.
4. Match model size to the job. On this machine, the smaller Gemma E2B beat the larger E4B on latency and memory, so check whether you need the larger model at all.
5. Measure on your own hardware. These numbers come from an M1 MacBook Air on battery, and your results will differ.

Run your own benchmark before you commit to a model for your agent stack. It costs little and it can change which model you pick.

Originally published at memo.jimmyliao.net.