TPU v6e vs A100 80GB 2 for Gemma 4 31B on vLLM: 21 Benchmarks Show When Each Wins

Introduction

Why "speed" and "cost" of LLM inference matter more than ever

LLMs have moved from research demos to product surfaces, and the operational concern has shifted from accuracy to how fast and how cheap we can serve them. For chat-style interactions, TTFT (Time To First Token) is the metric users actually feel; for RAG, summarization, and code generation, TPOT (Time Per Output Token, the average inter-token time after the first one) and the server-wide throughput drive the bill.

Whether on-prem or on GCP / AWS / Azure, hardware choice can swing the cost of running the same model by multiples. That's exactly why a benchmark that says, for your workload, which placement to pick is increasingly valuable.

Why vLLM

vLLM has effectively become the de-facto serving engine for open-weight LLMs. We picked it for this benchmark because:

• PagedAttention removes KV-cache fragmentation. A virtual-memory-style paging scheme makes variable-length KV cache cheap to manage, which boosts concurrent request capacity.
• Continuous batching keeps the accelerator busy. Tokens — not requests — are the scheduling unit, so a freed slot is immediately filled by the next prefill. Throughput consistently dominates static batching.
• OpenAI-compatible API (/v1/chat/completions, /v1/completions) — drop-in for existing clients.
• Multi-accelerator support — the same code path runs on NVIDIA (Ampere/Hopper), AMD ROCm, and Cloud TPU (v5e, v6e). That's exactly what makes a fair TPU vs GPU comparison possible in a single article.
• Official Docker images (vllm/vllm-openai, vllm/vllm-tpu) eliminate dependency drift. We used them throughout this benchmark.

What this article covers

We use Gemma 4 31B Instruct, the latest dense model from Google, and serve it on GCP with vLLM, running vllm bench serve against:

• TPU v6e-4 (Trillium)
• NVIDIA A100 80GB × 2

Specifically:

• 3 input/output profiles (short / medium / long) × 7 request rates (1, 2, 4, 8, 16, 32, ∞) → 21 cases per platform.
• Three families of metrics: Throughput (req/s, tokens/s), Latency (TTFT, TPOT, ITL), Cost ($ / 1M output tokens).
• One important caveat: vLLM's Gemma 4 attention path has an assert that rejects fp8_e5m2 KV cache, and fp8_e4m3 is Hopper-only. So A100 ran with bf16 KV cache while TPU ran with fp8 — we'll be explicit about how that affects the numbers below.

Beyond the numbers, we also share the operational gotchas we hit on both sides — quota, capacity, image setup, error reading — so readers can budget for them next time.

Environment & Setup

Hardware

We followed vLLM's official Gemma 4 recipe, which lists the minimum requirement for the 31B dense variant as 80GB GPU with TP=2 or TPU v6e (Trillium) × 4 chips.

Class	GCP instance	Accelerator	Parallelism
GPU (A100)	a2-ultragpu-2g	A100 80GB × 2	tensor-parallel-size=2
TPU (Trillium)	v6e-4	TPU v6e × 4	tensor-parallel-size=4

H100 80GB×2 is on the wishlist for a follow-up (see Future work).

L4 (24GB) and TPU v5e (16GB) lack the memory for Gemma 4 31B and are not part of this comparison.

Software

We run vLLM through the official Docker images on both sides. Pip-based install is technically possible on GPUs, but on TPU the standard PyPI vLLM (CUDA build) gets pulled in and the server fails with Failed to infer device type. To keep operations symmetric, we put both platforms on Docker.

Item	GPU side	TPU side
Host OS	Ubuntu 24.04 LTS (DLVM common-cu129-ubuntu-2404-nvidia-580)	Ubuntu 22.04 LTS (TPU runtime v2-alpha-tpuv6e)
Host Python	3.12	3.10
CUDA / TPU stack	CUDA 12.9 + NVIDIA Driver 580	libtpu (v6e), torch_xla (bundled in image)
vLLM Docker image	vllm/vllm-openai:latest	vllm/vllm-tpu:gemma4 (per Gemma 4 recipe)

GPU-side launch:

sudo docker run -itd --name gemma4-gpu \
  --gpus all --ipc=host --shm-size=16g \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e HF_TOKEN="$HF_TOKEN" -p 8000:8000 \
  vllm/vllm-openai:latest \
    --model google/gemma-4-31B-it \
    --tensor-parallel-size 2 \
    --max-model-len 16384 \
    --gpu-memory-utilization 0.90 \
    --kv-cache-dtype auto \
    --host 0.0.0.0 --port 8000

TPU-side launch (per the official recipe; we put vllm serve first because the entrypoint is a pass-through):

sudo docker run -itd --name gemma4-tpu \
  --privileged --network host --shm-size 16G \
  -v /dev/shm:/dev/shm \
  -e HF_TOKEN="$HF_TOKEN" \
  vllm/vllm-tpu:gemma4 \
    vllm serve google/gemma-4-31B-it \
      --tensor-parallel-size 4 \
      --max-model-len 16384 \
      --disable_chunked_mm_input \
      --host 0.0.0.0 --port 8000

KV cache dtype: not symmetric between GPU and TPU

We wanted to align the KV cache dtype across platforms, but the combination we wanted wasn't possible:

1. fp8_e4m3 (= fp8e4nv) is Hopper-only ; on A100 it fails Triton compilation.
2. fp8_e5m2 is rejected by vLLM's Gemma 4 attention implementation with assert kv_cache_dtype in {"fp8", "fp8_e4m3"}.

→ A100 was forced to --kv-cache-dtype auto (= bf16). TPU v6e auto-enables fp8_e5m2 (we saw the log line Automatically using fp8_e5m2 for FP8 KV cache on TPU v6e).

The practical impact: A100's per-request KV cache is roughly 2× larger than TPU's , which roughly halves its concurrent capacity. Read the GPU numbers below as the bf16-KV-cache number, not as an upper bound for A100. The clean FP8-on-FP8 comparison is on H100 — we're moving that to Future work.

For reproducibility, we capture docker image inspect output (RepoDigests) alongside each run.

Model

• google/gemma-4-31B-it (31B dense, instruction-tuned, Apache 2.0, 256k context)

Gemma 4 is officially supported by vLLM, so the same code path serves it on both TPU and GPU.

Dataset

We use vLLM's built-in --dataset-name random so we can pin input/output token lengths exactly. This guarantees identical workload shapes across platforms.

Profile	Input tokens	Output tokens	Use case
short	1024	256	Chat replies
medium	4096	512	RAG / mid-length generation
long	8000	1000	Long-form summarization, code generation

Methodology

How we drove the load

We use vLLM's official vllm bench serve CLI as the load generator. The server runs in one process (vllm serve); the bench client runs separately.

Server start (example, H100×2 with TP=2):

vllm serve google/gemma-4-31B-it \
  --tensor-parallel-size 2 \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.90 \
  --kv-cache-dtype fp8

--kv-cache-dtype fp8 is the recipe-recommended option that cuts KV cache memory by ~50% (when supported).

Bench run (example):

vllm bench serve \
  --model google/gemma-4-31B-it \
  --dataset-name random \
  --random-input-len 1024 --random-output-len 256 \
  --request-rate 8 --num-prompts 1000

The TPU side runs the same model, dataset, and lengths — only --tensor-parallel-size changes to 4.

Sweep

For each platform we sweep 3 input/output profiles × request-rate ∈ {1, 2, 4, 8, 16, 32, inf}, so 21 cases. inf corresponds to offline maximum-throughput measurement. Each run is preceded by a single warmup round; results are saved as JSON and uploaded to GCS.

Metrics

Three families:

1. Throughput

• Output throughput (tokens/s): server-wide output tokens generated per second
• Request throughput (req/s): completed requests per second

2. Latency

• TTFT: time from request to first token
• TPOT: average inter-token time after the first token
• E2E p50/p95/p99: full-request latency percentiles

3. Cost

We compute "cost per 1M output tokens" in $ from the GCP on-demand price:

$/1M tokens = (instance_$_per_hour / 3600) × total_seconds / total_output_tokens × 1_000_000

We use on-demand prices (no Spot) at the time of measurement.

Results

For each profile we fix --num-prompts 1000 and sweep request rate over 1 / 2 / 4 / 8 / 16 / 32 / ∞. Numbers below are taken from the raw JSON: request_throughput, output_throughput, median_ttft_ms, median_tpot_ms.

Summary: saturation throughput

For each profile, we take the maximum Output throughput across the 7 rates as the "saturation" point. TPU wins on short, A100 wins on medium and long.

Output throughput vs. request rate

X axis is request rate (req/s); ∞ is --request-rate inf (1000 prompts dispatched at once). On short, TPU scales linearly from rate=4 and lands above 1,000 tokens/s. On medium and long, both platforms saturate at rate=1 and the curves are flat above it.

TPOT (per-output-token time) vs. request rate

TPOT is lower (= faster) on TPU across every profile and every rate. This is the metric that drives the perceived smoothness of streaming responses.

TTFT (time to first token) vs. request rate (log scale)

The Y axis is logarithmic. At low rates both platforms stay in the millisecond range; once saturated, queues stretch into seconds and minutes. The high-rate end of short and the entire long profile show this clearly.

Short profile (input 1024 / output 256)

Rate	TPU req/s	A100 req/s	TPU out_tok/s	A100 out_tok/s	TPU TTFT p50 (ms)	A100 TTFT p50 (ms)	TPU TPOT p50 (ms)	A100 TPOT p50 (ms)
1	1.00	0.99	255	254	123	249	17.6	33.6
2	1.98	1.97	508	504	135	291	23.6	55.4
4	3.76	2.56	963	656	6,634	86,709	46.2	116.7
8	3.82	2.88	978	738	64,592	99,110	45.1	115.4
16	3.89	2.89	995	740	90,980	130,163	45.0	114.7
32	3.92	2.89	1,004	740	104,790	145,809	44.7	114.7
∞	3.91	2.88	1,002	737	120,783	162,543	44.7	114.8

• Saturation: TPU lands at ~3.9 req/s / ~1,000 tokens/s, A100 at ~2.9 req/s / ~740 tokens/s. TPU is 1.35–1.36× higher.
• A100 spikes TTFT to 86 seconds at rate=4, indicating it has hit the concurrency ceiling (bf16 KV cache).
• TPOT is roughly half on TPU at saturation (44.7 ms vs 114.8 ms). TPU is 2.55× faster.

Medium profile (input 4096 / output 512)

Rate	TPU req/s	A100 req/s	TPU out_tok/s	A100 out_tok/s	TPU TTFT p50 (ms)	A100 TTFT p50 (ms)	TPU TPOT p50 (ms)	A100 TPOT p50 (ms)
1	0.78	0.83	400	425	132,484	94,063	29.5	153.5
2	0.79	0.84	402	430	375,386	332,822	29.7	153.0
4	0.79	0.84	403	430	498,994	458,066	29.8	153.0
8	0.79	0.84	403	430	561,402	521,088	29.8	153.1
16	0.79	0.84	403	430	592,751	552,527	29.8	153.1
32	0.79	0.84	403	430	608,386	568,216	29.7	153.1
∞	0.79	0.84	403	429	624,698	585,161	29.7	153.1

• A100 wins by 6–7% on saturation throughput (430 vs 403 tokens/s). Both saturate at rate=1; raising the rate only stretches TTFT.
• TPOT: TPU is 5.1× faster (29.7 ms vs 153.1 ms).

Long profile (input 8000 / output 1000)

Rate	TPU req/s	A100 req/s	TPU out_tok/s	A100 out_tok/s	TPU TTFT p50 (ms)	A100 TTFT p50 (ms)	TPU TPOT p50 (ms)	A100 TPOT p50 (ms)
1	0.264	0.381	264	381	1,384,448	773,158	22.4	129.3
2	0.263	0.381	263	381	1,639,398	1,024,209	22.4	129.3
4	0.263	0.380	263	380	1,764,755	1,150,795	22.3	129.4
8	0.263	0.381	263	381	1,827,451	1,213,094	22.3	129.3
16	0.263	0.380	263	380	1,859,960	1,244,920	22.3	129.4
32	0.263	0.381	263	381	1,875,266	1,259,711	22.3	129.3
∞	0.263	0.380	263	380	1,891,177	1,276,151	22.3	129.4

• A100 leads saturation throughput by 1.44× (380 vs 263 tokens/s). Long-input prefill performance dominates here.
• A100's TTFT is 35–40% lower — long prefills finish faster, so the queue drains faster.
• TPOT: TPU is 5.8× faster (22.3 ms vs 129.4 ms).

Combined summary

Saturation values from each profile in one place:

Profile	Saturation req/s (TPU / A100)	Saturation out_tok/s (TPU / A100)	TPOT p50 (TPU / A100)
short (1024/256)	3.92 / 2.89	1,004 / 740	44.7 / 114.8 ms
medium (4096/512)	0.79 / 0.84	403 / 430	29.7 / 153.1 ms
long (8000/1000)	0.263 / 0.381	263 / 380	22.3 / 129.4 ms

Throughput's winner switches with input/output length, while TPOT is 2.55–5.8× faster on TPU across every profile.

Cost estimate ($ / 1M output tokens)

The relationship is straightforward:

$/1M tokens = (instance_$_per_hour / 3600) × 1,000,000 / output_tok_per_s
            = instance_$_per_hour / 3.6 / output_tok_per_s

Plug in the saturation throughput and your hourly rate:

Profile	TPU v6e-4 (out_tok/s)	A100 80GB×2 (out_tok/s)
short	1,004	740
medium	403	430
long	263	380

If TPU v6e-4 and A100 80GB×2 are priced similarly per hour (which is roughly the case in our regions), TPU is cheaper for short and A100 is cheaper for medium/long. Replace the rates with your own figures for an exact answer.

Discussion

The throughput winner switches with input/output length

The most striking finding is that TPU wins for short, A100 wins for long. The cause is that LLM inference has two phases — prefill (process the entire prompt at once) and decode (one token at a time) — and the balance shifts with input/output length:

• short (1024 in / 256 out): 256-token decode dominates wall time. TPU's per-token decode efficiency wins.
• medium (4096 in / 512 out): 4K prefill and 512 decode are roughly balanced. A100 edges ahead.
• long (8000 in / 1000 out): 8K prefill dominates. A100's prefill compute pays off.

TPOT is dramatically faster on TPU across every profile

Profile	TPOT ratio (A100 / TPU)
short	2.55×
medium	5.15×
long	5.80×

This shows TPU v6e's per-token decode efficiency is consistently strong. For streaming UX (where each token shows up immediately on screen), this is the metric your users actually feel. The "fluidity" of generated text is significantly smoother on TPU.

Caveat: A100 ran with bf16 KV cache

When reading the throughput numbers, remember that A100 was on bf16 KV cache (see the Software section): fp8_e4m3 is Hopper-only and fp8_e5m2 is rejected by vLLM's Gemma 4 attention assert. So A100 was running on roughly half the concurrent capacity of TPU's fp8 setup — and still beat TPU on medium and long throughput. On Hopper (H100) with fp8_e4m3, the gap could grow further. Conversely, TPU v6e — running aggressive fp8 KV compression — still beats A100 on short and on TPOT across the board.

Workload-based recommendation

Mapping the numbers to product use cases:

Use case	Profile	Recommendation	Rationale
Chat (interactive UX)	short (~1K in / ~256 out)	TPU v6e-4	Higher saturation throughput and lower TPOT → fast, smooth replies
RAG / mid-length generation	medium (~4K in / ~512 out)	Either	Saturation throughput is close. TPU if TPOT matters; A100 if cost is closer
Long-form summarization / code	long (~8K in / ~1K out)	A100 80GB × 2	1.44× higher throughput on long prefill — though TPU still has smoother token streaming
Streaming UX above all	any profile	TPU v6e-4	TPOT 2.5–5.8× faster than A100

Operational notes: where TPU and GPU each tripped us up

We hit multiple snags on the way to a clean run. Interestingly, the layers that broke on TPU and on GPU were quite different.

TPU side

• Don't install vLLM via pip on TPU. PyPI's plain vllm is the CUDA build; the server fails at startup with Failed to infer device type. The official Docker image (vllm/vllm-tpu:gemma4) is far simpler.
• Image entrypoint quirks. vllm/vllm-tpu:gemma4 uses a pass-through entrypoint, so the first arg has to be a runnable command. We put vllm serve <model> first; passing --model X directly produces bash: exec --: invalid option.
• Pick the right TPU runtime. v2-alpha-tpuv6e. Confirm with gcloud compute tpus tpu-vm versions list --zone=$ZONE. TPU 7x (Ironwood) wasn't accessible from our project — application is in flight.
• TPU v6e capacity is tight. Even with unlimited quota, Insufficient capacity was the most common failure; us-central1-{a,c} and us-east5-a were all capacity-constrained, and we eventually landed on us-east5-b. Queued Resource (QR) is a practical fallback if every zone fails on the first try.

GPU side

The GPU side hit fewer accelerator-specific issues; most of the friction was Linux distro and GCP resource management.

• Fresh GCP projects start with zero GPU quota. The project we used had NVIDIA_A100_80GB_GPUS and NVIDIA_H100_80GB_GPUS at 0 in every region. We submitted three quota requests — the first two went out for Committed NVIDIA H100 GPUs (CUD), which were approved but useless for on-demand. We then requested both plain NVIDIA X 80GB GPUs (without the Committed or Preemptible prefix) and the new umbrella GPUS_PER_GPU_FAMILY (with gpu_family=NVIDIA_H100); we're still not sure exactly which combination is the bare minimum and would love expert input on this.
• A100 had capacity issues too. Both us-east5-{a,b} returned ZONE_RESOURCE_POOL_EXHAUSTED. We followed GCP's hint and ended up in us-central1-c. (The TPU and GPU ended up in different regions; since the bench loops through docker exec on the same VM, we believe this doesn't affect comparison fairness.)
• DLVM common-cu129-ubuntu-2404-nvidia-580 does not include Docker. It ships NVIDIA Driver 580 and NVIDIA Container Toolkit 1.17.8, but Docker engine itself you install yourself: sudo apt install -y docker.io followed by sudo nvidia-ctk runtime configure --runtime=docker.
• A fresh VM has its apt held by unattended-upgrades. sudo apt install gives Could not get lock for the first few minutes. Our setup script now waits on the lock before proceeding.
• Ubuntu 24.04 enforces PEP 668. pip install --user is blocked by externally-managed-environment. Tools like the HF CLI now go in a dedicated venv.
• huggingface-cli is deprecated → hf CLI. As of huggingface_hub 1.12, huggingface-cli prints a "use hf instead" warning and stops working. The replacements are hf auth login --token and hf download REPO_ID.
• HF cache permission collision. Once Docker (running as root) writes into ~/.cache/huggingface, the host user can't always write back: [Errno 13] Permission denied: ...lock. Fixed with sudo chown -R $USER:$USER ~/.cache/huggingface.
• A100 + Gemma 4 can't use FP8 KV cache. Detailed above; the practical resolution is --kv-cache-dtype auto (= bf16).
• Read vLLM v1 errors at the worker level. Engine core initialization failed. See root cause above. Failed core proc(s): {} is a messenger, not the actual exception. The real exception is in lines prefixed with (Worker_TP*). We extracted them with docker logs gemma4-gpu | grep '(Worker_TP'.

Cross-platform takeaways

• Don't paste long shell commands through chat / IDE buffers. Backslash continuations and heredocs corrupt easily. Author them as files locally, scp them, then bash them.
• Run long jobs under nohup or tmux/screen from the start. A single ssh disconnect cost us more than an hour of progress and accelerator time.

TPU breaks at the accelerator-specific layers (image, runtime, arg parsing, capacity). GPU breaks at the Linux distro and GCP-resource layers (quota, Docker, pip, hf). Plan for a half-day to one-day of buffer per platform on a fresh project.

Conclusion

Summary of findings

We benchmarked Gemma 4 31B on TPU v6e-4 and A100 80GB × 2 with vLLM, running 21 cases (3 profiles × 7 rates) on each side. Highlights:

• Throughput winner depends on input/output length. TPU wins short. A100 wins medium and long.
• TPOT is uniformly faster on TPU by 2.5–5.8×. For streaming UX, TPU has a real edge.
• A100 ran with bf16 KV cache, not FP8. It still won on medium and long. On Hopper with fp8_e4m3, the gap could grow further.
• The cleanest split: chat → TPU; long-form generation → A100.
• Operationally, TPU and GPU break in different layers. Budget time accordingly on fresh projects.

For middle-of-the-road dense LLMs in the ~30B class on GCP, this hopefully gives a concrete reference for choosing TPU or GPU based on your actual workload shape.

Future work

• H100 + FP8 KV cache for a fair fight. A100 was forced to bf16; H100 with fp8_e4m3 would let us put GPU on the same FP8 footing as TPU and rerun the sweep.
• TPU 7x (Ironwood). 192 GiB HBM/chip — six times TPU v6e. Currently allowlist-only, but a follow-up benchmark is planned once it's broadly available.
• Quantized weights. AWQ, GPTQ, and FP8 weight quantization for accuracy/throughput tradeoffs.
• Larger and MoE models. Gemma 4 26B-A4B (MoE) and bigger dense models with quantization.

Appendix: Smoke test on the short profile

Before the main sweep, we ran a one-off smoke test on the short profile (input 1024 / output 256), submitting 1000 prompts with --request-rate inf-equivalent dispatch (Maximum request concurrency: None, Peak concurrent requests: 1000). The numbers aren't part of the official sweep but give a quick pre-flight comparison.

The authoritative numbers are in the Results section above. The smoke test is a single one-shot run right after Docker container startup, while the main sweep includes a warmup round before each of the 21 cases. The post-warmup A100 numbers come out slightly better (rate=∞: req/s 2.78 → 2.88, out_tok/s 712 → 737, median TTFT 174,897 → 162,543 ms; TPU is virtually unchanged). Treat the appendix as a "yes, both platforms are running the same configuration" sanity check rather than a benchmark snapshot.

Metric	TPU v6e-4 (FP8 KV)	A100 80GB × 2 (bf16 KV)	Ratio (TPU / A100)
Successful requests	1000	1000	—
Failed requests	0	0	—
Benchmark duration (s)	256	359	—
Request throughput (req/s)	3.91	2.78	1.40×
Output token throughput (tok/s)	1001	712	1.41×
Total token throughput (tok/s)	5004	3561	1.41×
Mean TTFT (ms)	121,590	176,526	TPU 1.45× faster
Median TTFT (ms)	120,927	174,897	TPU 1.45× faster
P99 TTFT (ms)	240,983	348,870	TPU 1.45× faster
Mean TPOT (ms)	46.8	112.3	TPU 2.40× faster
Median TPOT (ms)	44.8	114.7	TPU 2.56× faster
P99 TPOT (ms)	79.3	150.3	TPU 1.90× faster

Observations:

• Throughput is ~1.4× higher on TPU. A100 is on bf16 KV cache (~half the concurrency of TPU's fp8), so this isn't a pure hardware gap.
• TPOT is ~2× faster on TPU. KV cache size doesn't directly affect TPOT, so this is closer to a clean compute comparison and v6e looks strong. The main sweep confirms the pattern.
• TTFT is in the 100-second range for both platforms. This is --request-rate inf with 1000 prompts thrown in at once — queue-dominated. In real Poisson traffic at lower rates, TTFT shrinks dramatically; the main sweep shows the realistic numbers at rate=1–32.

References

• vLLM Documentation

• vLLM Gemma 4 recipe

• Gemma 4 31B Instruct on Hugging Face

• Cloud TPU v6e (Trillium) docs

• LLM Inference Hardware Benchmark (living hub) — this campaign indexed alongside all our other inference measurements