Benchmarking Gemini 2.5 Flash vs 3.1 Flash-Lite vs Gemma 4 with LLM judge (Claude Fable 5)

Cross-posted from the IO reader blog, where the full version includes all 36 unedited transcripts side by side.

With gemini-2.5-flash scheduled to retire on October 16, we needed to audition replacements for a production workload that generic leaderboards say nothing about: twelve distinct "reading lens" prompts — literary close reading, Socratic questioning, EN→KO translation under a hard register constraint, vocabulary coaching — that our iOS reading app serves through Firebase AI Logic.

So we ran the audition the unfashionable way: one adversarial stimulus, every production prompt verbatim, three models, and a human reading + LLM judge (Claude Fable 5).

Method

Stimulus. The closing paragraph of Melville's "Whiteness of the Whale" chapter — chosen to break things: archaic vocabulary (subtile, palsied, charnel-house), one enormous periodic sentence, nineteenth-century natural philosophy, and a live metaphysical argument. If a model holds this paragraph steady through twelve different lenses, it holds almost anything a reader will meet.

Models.

• gemini-2.5-flash — the incumbent engine
• gemini-3.1-flash-lite — its smaller, newer successor candidate
• gemma-4-26b-a4b-it — open-weights reasoning contender

Protocol. Production prompts verbatim as a single user turn; temperature 0.4; thinkingBudget: 0 on the Gemini models; reasoning left on for Gemma (its serving default). One generation per cell — 12 × 3 = 36 calls — with latency and token counts taken from the API's own usageMetadata.

Results

Model	Runs	Median	P90	$/1M in·out	Measured billed ¢/reply
gemini-2.5-flash	12/12	1.7 s	2.8 s	0.30 · 2.50	0.057¢
gemini-3.1-flash-lite	12/12	2.2 s	3.0 s	0.25 · 1.50	0.038¢
gemma-4-26b-a4b-it	12/12	49.6 s*	126.8 s*	0.15 · 0.60	0.150¢

*Shared research endpoint with reasoning enabled — compare Gemma's quality here, not its speed. "Billed ¢/reply" is computed from each model's measured median prompt and output tokens at list price, including hidden reasoning tokens, which are billed as output.

Finding 1: Format compliance was 36/36 — on every model

Word caps, fixed output structures, question-only constraints, and the Korean formal-register rule (해라체, never conversational endings) all held across every generation. A prompt contract, it turns out, is a property of the contract — not of the model behind it. That's what makes engines swappable without the product changing shape.

Finding 2: The smallest, newest model read best

Gemini 3.1 Flash-Lite — 40% cheaper on output than the incumbent — kept producing the sharpest specific observations. It was the only model to name Melville's "colourless, all-colour" paradox outright, and it described the prose as "a rhetorical trap" whose cumulative clauses leave the reader "as intellectually blinded as the wretched infidel."

The cleanest contrast: one of our lenses must end every response with a single unsettling question. Same paragraph, same instructions, three minds —

"Does the terror of whiteness truly stem from its objective qualities, or from humanity's ingrained need to project meaning and warmth onto a fundamentally indifferent cosmos?" — gemini-2.5-flash

"If color is a lie, is truth merely the blindness of the void?" — gemini-3.1-flash-lite

"If the truth is a shroud, is the only way to remain sane to remain deceived?" — gemma-4-26b

The incumbent is thorough and slightly padded. The understudy is compressed and exact. The open-weights model writes like it's been saving that sentence all week.

Finding 3: Sticker price is not delivered price

Gemma 4 26B has the lowest list price on the bench and the highest measured cost per reply — roughly 4× Flash-Lite. The gap is hidden reasoning: with its default thinking mode on, it deliberated for 5,000–10,000 unseen tokens per answer, and reasoning tokens bill as output.

If you're comparing models on list price, check usageMetadata.thoughtsTokenCount first. For self-hosted batch pipelines Gemma's quality is real and the economics change entirely; for interactive serving through a metered API, list-price comparisons are fiction.

Limitations

One generation per cell — a structured qualitative reading, not a statistical claim. Judging wasn't blind. Gemma's latency reflects a shared research endpoint, not managed serving. And one passage is one genre; the next run adds an analytical text, a modern novel, and poetry.

The full transcripts

All 36 outputs — unedited, side by side, including both Korean translations and the answers we found weakest — are in the full research note:

→ ioreader.app/blog/the-whiteness-test

Disclosure: IO is our app; the benchmark used its production prompts. Curious how others are accounting for thinking-token billing when comparing reasoning models — comments welcome.