DeBERTa on ZeroGPU vs Gemini Flash: 100% vs 92% on IAB Tier-1

#ai #machinelearning #llm

Model benchmarks read best when the task is narrow, the protocol is explicit, and every percentage sits next to a baseline. This note follows that approach: one task, fixed labels, transparent scoring, and clear caveats where interpretation depends on sample size or infrastructure.

We compare a hosted specialist classifier, DeBERTa-v3-small on ZeroGPU's Responses API, to a general Gemini model (gemini-flash-latest) on identical article snippets and the same closed IAB Content Taxonomy 3.1 Tier-1 label set.

The goal is not to declare a universal winner. It is to show what happens when both systems are evaluated on the same narrow production task.

What we are measuring

Task: Map short editorial text to exactly one Tier-1 IAB category (18-label closed set).

Primary metric: Tier-1 accuracy, the fraction of examples where the predicted label matches the reference after normalization (case, spacing, punctuation, "&" vs "and").

Baselines on the same 50 examples

Baseline	Accuracy (%)	Δ vs majority
Majority class (Technology & Computing)	10	0
Uniform random (expected)	10	0

These baselines define the floor: results near 10% are not informative for production.

How we run it on ZeroGPU

Evaluation is executed as standard API inference with consistent prompts and scoring:

Transport: HTTP POST to /v1/responses with standard authentication
Model: deberta-v3-small (hosted classifier SKU)
Prompt shape: chat-style input with
- system message listing all allowed Tier-1 labels and requiring a single verbatim label
- user message containing the article text

Each example is scored against a fixed 50-sample golden set.

Latency is client-observed end-to-end wall time per request (network, scheduling, inference). This is not a synthetic microbenchmark.

How we run Gemini (matched setup)

Each snippet is sent to Gemini generateContent as a single text block that embeds:

the same 18-label list
the same instruction: "reply with exactly one label, verbatim"

This aligns the task definition despite different APIs.

Latency is measured identically: client-observed round-trip per request. Throughput (tokens/sec) is omitted where usage metadata is unavailable.

Results (same 50-sample golden set)

Model	Provider	Accuracy (%)	Δ vs majority	Avg latency (ms)	Tokens/sec (est.)
deberta-v3-small	ZeroGPU	100	+90	1324	82
gemini-flash-latest	Google Gemini	92	+82	27027	n/a

Accuracy and latency move together here: DeBERTa reaches perfect accuracy on this slice with ~1.3s average latency, while Gemini Flash reaches 92% at ~27s under the same evaluation loop.

For a single-label routing workload, that latency gap is operationally significant.

Cost (list pricing, before discounts)

Benchmark notes should include price context alongside accuracy.

Token list rates (USD per 1M tokens):

Offering	Input	Output
deberta-v3-small on ZeroGPU	0.05	0.40
Gemini Flash (standard tier)	0.30	2.50

The Gemini column reflects Flash-class pricing from Google's published API rates. The alias gemini-flash-latest maps to the active Flash SKU in your account.

For Tier-1 routing at scale, classifier SKUs typically produce lower total cost of ownership than general-purpose generation models, primarily due to output token pricing.

Always confirm live pricing before budgeting.

Why CPU-first serving can outperform GPU here

For narrow classification workloads like Tier-1 IAB routing, CPU-first serving is often the practical choice:

Small models run efficiently without GPU batching
Online traffic arrives in low-batch patterns
The task is fixed-label discrimination, not generation
Predictable latency matters more than peak throughput

For this benchmark shape, CPU is not a fallback. It is often the faster path.

Takeaway

For fast, deterministic Tier-1 routing, a hosted classifier is often the right abstraction: higher accuracy, lower latency, and simpler cost structure.

For open-ended generation on the same text, a Gemini-class model remains the appropriate tool. This benchmark isolates a task where it is not.