DEV Community: Sho Tanaka (tsho)

Turning Qwen2.5-0.5B into a JSON API with SFT — 23% 100% on a Free T4

Sho Tanaka (tsho) — Wed, 19 Aug 2026 09:00:00 +0000

I Tried Post-Training and All I Got Was "systemsystem"

I followed the TRL tutorial to fine-tune a small model with SFT (supervised fine-tuning). Training ran to completion. But no matter what I asked the trained model, the answer was:

systemsystemsystemsystemsystem...

Was the model broken? Was my code wrong? I couldn't even tell which — if you've tried SFT, maybe this sounds familiar.

It turned out I hit that exact symptom twice, from two unrelated causes : once because training genuinely diverged, and once because training had succeeded perfectly and my verification code was wrong. Telling those apart is most of what this article is about.

This is the field log of getting from there to turning Qwen2.5-0.5B into a model that answers every question with nothing but JSON.

Item	Result
Model	Qwen2.5-0.5B base
Method	SFT (implemented with LoRA)
Framework	TRL + PEFT
GPU	Colab T4 ×1 (free tier)
Training time	26 minutes (1,538 s)
Valid-JSON rate	23% → 100%
Valid JSON and correct category	0% → 96%
Training data	8,705 single-turn samples (no_robots)
Evaluation	30 held-out questions

A note on the title: SFT and LoRA are not alternatives to each other. SFT is the objective — train on demonstrations of the behavior you want. LoRA is a parameterization — store the weight update in a small low-rank adapter instead of touching all the weights. Everything below is caused by SFT; LoRA is just where the result is kept. (Why LoRA and not full fine-tuning is itself Trap 4.)

This is part 1 of a post-training series; part 2 covers RM+PPO / DPO.

Setup: Why a 0.5B Base Model

Item	Value
Model	`Qwen/Qwen2.5-0.5B` (base — not `-Instruct`)
Data	`HuggingFaceH4/no_robots` (train 9,500 / test 500; `messages` + `category` columns)
Libraries	TRL 1.10.0, transformers 5.13.1, peft 0.19.1, datasets 5.0.1, accelerate 1.14.0, torch 2.11.0+cu128
Environment	Colab T4 ×1, pinned to a single GPU

Three reasons for this configuration:

Without a base model there is no "before." -Instruct models have already been through SFT, so they can't show you what SFT changes.
GGUF and MLX are inference-only formats — you can't train them. You need safetensors.
Full SFT of a 30B-class model needs ~480 GB of VRAM , while a 0.5B model still exposes the same operational failure modes — and a free T4 shows you all of them.

Those library versions matter more than usual here: two of the seven traps below are pure version drift , and they will look different (or disappear) on other versions.

Seven Traps, in Order

Chronological. Each one is error → cause → fix.

Trap 1: `KeyError: 'completion'`

KeyError: 'completion'

Died the moment training started. Cause: no_robots carries a prompt column alongside messages, and TRL misdetected the dataset as prompt-completion format. One-line fix:

train = ds["train"].select_columns(["messages"])

Trap 2: Output Collapses to "systemsystemsystem…"

The one from the intro — the real one. Cause: full fine-tuning directly on fp16-loaded weights diverged. The T4 has no native bf16, so fp16 is the tempting choice, but updating fp16 weights directly is numerically fragile; the model collapses onto its most frequent token.

Fix: load in fp32 and let mixed precision be handled by the trainer, which keeps an fp32 master copy.

model = AutoModelForCausalLM.from_pretrained(MODEL, dtype=torch.float32)
cfg = SFTConfig(fp16=True, ...)

Trap 3: `RuntimeError` — Tensors on cuda:1 and cuda:0

RuntimeError: tensors ... cuda:1 different from cuda:0

I tried Kaggle as well as Colab. Kaggle gives you two T4s, HF Trainer silently wraps the model in nn.DataParallel, and it dies on a device mismatch. A 0.5B model doesn't need two GPUs. Fix — but it only takes effect before import torch, which means a kernel restart if you've already imported:

import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
import torch

Trap 4: Full Fine-Tuning Collapses Again — Why I Switched to LoRA

Even with Trap 2 fixed, full fine-tuning of this small model stayed extremely sensitive to learning rate and precision, and it collapsed again. That was the turning point: I dropped full FT for LoRA (r=16, alpha=32, lr=2e-4). With the base weights frozen and only 8.8M adapter parameters training (1.75% of the model), the instability went away and never came back.

Caveat I want to be honest about: I never got a clean full-FT run to compare against. So "LoRA is more stable here" is what I observed, not a controlled A/B.

Trap 5: Forgot `model.eval()` — a Fake Collapse

The one that made me go pale. After switching to LoRA, generation said systemsystem again.

Training had actually succeeded. I had forgotten model.eval(), so greedy decoding ran with dropout still active. I isolated it by varying two things independently:

`training`	`use_cache`	output
True	False	`A systemsystemsystem…`
True	True	`A systemsystemsystem…`
False	False	`A cache is a data structure that stores data…`
False	True	`A cache is a data structure that stores data…`

Only training mattered. Meanwhile the training logs were healthy the whole time — entropy 2.63 → 2.41 (a real collapse drives it toward zero), grad_norm 1.17 → 0.45, token accuracy 0.469 → 0.517.

Diagnose collapse from the training logs, not from the symptom. The symptom is ambiguous; entropy and grad_norm are not.

was_training = model.training
model.eval()
try:
    ...generate...
finally:
    model.train(was_training)

Trap 6: `torch.cuda.is_bf16_supported()` Returns True on a T4

Recent PyTorch changed this function's default to including_emulation=True, so a T4 answers "yes" and your code happily selects an emulated bf16 that is slower with no benefit:

capability: (7, 5)
is_bf16_supported(): True
is_bf16_supported(including_emulation=False): False

Check the hardware generation instead — it's stable across versions:

use_bf16 = torch.cuda.get_device_capability(0)[0] >= 8 # Ampere or later

Trap 7: Two Version-Drift Errors (`BatchEncoding`, `torchao`)

apply_chat_template() returns a BatchEncoding, not a tensor.

AttributeError: ... 'inputs.shape'

Pass return_dict=True, call generate(**enc), and take the prompt length from enc["input_ids"].shape[1].

Colab's preinstalled torchao is too old for recent transformers.

ImportError: Found an incompatible version of torchao. Found version 0.10.0,
but only versions above 0.16.0 are supported

This fires at import time, before any training. If you're not quantizing, remove it; if you are, upgrade it. Either way restart the session afterwards , or the preinstalled version stays loaded:

pip uninstall -y torchao # or: pip install -U torchao

What the trap list actually shows

I originally wanted to claim "most of these were verification-side problems." Lining them up, that isn't true. Two of the seven — Trap 5, and the BatchEncoding half of Trap 7 — were cases where training had genuinely succeeded and only my verification code was stale. The rest failed loudly and early: Traps 1 and the torchao half of 7 died before a single training step, and Traps 2/3/4 were real training failures.

The honest lesson is narrower but still worth having: when SFT "fails," check whether it failed at all. Two of my seven traps cost me hours on a model that was already working.

Experiment 1: What the Format Change Actually Looks Like

One epoch of SFT on no_robots, LoRA, 33 minutes on a T4. Same prompt, before and after — that comparison matters, because on some prompts the base is already better than you'd assume.

"Give me 3 tips for running a 10k race."

BEFORE (base):

Sure, here are three tips for running a 10k race:

1. Start early: It's important to start running early in the morning or early in
   the day to allow your body to adjust to the increased intensity of the race...
2. Listen to your body: Pay attention to...

AFTER (SFT):

1. Start early. If you start too late, you will not be able to run the full 10km.
   Start early and finish early.
2. Don't run too fast. You should be able to run the full 10km in 45 minutes or less...
3. Don't run too long...

Note what did not happen: the base already produced a numbered list. What SFT removed was the preamble (Sure, here are three tips for running a 10k race:) — the model now answers directly, in the style of the training data. That's a real change, and a smaller one than "it learned to make lists."

"What is the capital of France?" — this one is sharper:

BEFORE (base):

The capital of France is Paris. navigationOptions
navigationOptions
You are a helpful assistant.icode
icode
What is the capital of France?icode ...

AFTER (SFT):

The capital of France is Paris. It is the largest city in Europe and the
third-largest city in the world. It is also the seat of the French government
and the seat of the French parliament. It is also the seat of... (repeats)

The base answers correctly and then falls out of the assistant role entirely , echoing chat-template fragments. After SFT it stays in role from start to finish — and states two confident falsehoods (Paris is neither the largest city in Europe nor the third largest in the world). Form improved; factual reliability did not, and arguably got worse-looking because the errors are now delivered fluently.

"Explain what a KV cache is." — both loop, before and after. no_robots contains no KV-cache knowledge, and one epoch of SFT on it adds none.

The part I got wrong at first: SFT made stopping worse

My first read was "the leftover repetition is just a 0.5B + greedy-decoding artifact, not an SFT problem." I measured it, and that's only half right.

I generated 256 tokens for 25 held-out prompts and recorded two things separately: whether the model emitted the terminator <|im_end|>, and whether generation stopped at all.

| Model | Decoding | Stopped | Emitted <|im_end|> | distinct-3 |
| --- | --- | --- | --- | --- |
| base | greedy | 20% | 0% | 0.588 |
| base | greedy + repetition_penalty 1.1 | 40% | 0% | 0.781 |
| base | sampling (T=0.7, top_p=0.9) | 24% | 0% | 0.680 |
| SFT | greedy | 0% | 0% | 0.337 |
| SFT | greedy + repetition_penalty 1.1 | 0% | 0% | 0.700 |
| SFT | sampling | 0% | 0% | 0.534 |

Two separate phenomena, and they behave differently:

Repetition is decode-fixable. repetition_penalty moves distinct-3 from 0.337 to 0.700. My original claim holds here.
Termination is not , and SFT made it strictly worse: the base stopped 20–40% of the time; after SFT, never — under every decoding setting I tried.

Why? I looked at the terminal position of reference answers under teacher forcing:

| At the position whose target is <|im_end|> | base |
| --- | --- |
| P(<|im_end|>) | ~0 (below 1e-5) |
| Rank of <|im_end|> | 118,800 out of ~152,000 |
| P(<|endoftext|>) | 0.1006 |
| argmax was <|endoftext|> | 7 / 30 cases |

The base model's terminator is <|endoftext|>, and it barely knows <|im_end|> exists — despite shipping a chat template that uses it. Shipping a chat template and having been trained on it are different things, which I had conflated.

The training data ends every example with <|im_end|> and never contains <|endoftext|>. So SFT pushes down the only terminator the base knew, while failing to lift the new one from rank ~118,800 to rank 1 in one epoch of an 8.8M-parameter adapter. It didn't fail to teach stopping so much as break the stopping the model already had. That is my current explanation, and it's a hypothesis: I confirmed the base-side numbers but lost the VM before measuring the post-SFT side.

Two things I ruled out along the way: truncation at max_length=1024 affects only 1.7% of examples (163/9,500; median length 241 tokens), and training with assistant_only_loss=True — which concentrates the loss on assistant tokens, where the terminator lives — did not help (still 0% termination, and slightly more repetitive).

Experiment 2: Quantifying Success — a 0.5B JSON API

If SFT is good at teaching form, pick a task where form is measurable. Goal: make the model answer every question with nothing but {"category": "...", "answer": "..."}.

Labels reuse no_robots' existing category column, so additional data creation was zero. Restricted to single-turn conversations: train 8,705 / test 446, evaluated on the first 30 test questions.

Metric	BEFORE (base)	AFTER (SFT, 1 epoch)
Valid JSON extractable	23% (7/30)	100% (30/30)
Valid JSON and correct category	0% (0/30)	96% (29/30)

Measured run: 545 steps / 26 minutes (1,538 s) / train_loss 1.857 / mean_token_accuracy 0.653 / 2.10M tokens, on one T4.

How these metrics are defined , because it changes what they mean:

Valid JSON extractable = the first {...} found anywhere in the output parses and has both keys. It is deliberately lenient: preambles and trailing text don't disqualify a response. So 23% is not"23% of outputs were pure JSON."
The second row is a compound metric: a response only scores if it both parses and gets the category right. The base's 0% is therefore mostly a JSON failure, not a classification failure — 23 of its 30 answers never became scoreable. Don't read it as "the base cannot classify at all"; a base asked to emit only a bare category label would surely do better than zero.

With that in mind, the BEFORE 23% is the interesting number. Here's what one of those seven looked like:

Sure! Here's an example response for you:

{
"category": "Closed QA",
"answer": "I'm not sure what genre to check out."
}

Small talk, then a JSON object inside a markdown code fence , with a category that's wrong. The base can imitate the shape it was shown in the system prompt; it cannot execute the task (classify the request, then fill the structure), and it cannot suppress the surrounding chatter. My lenient parser counted this as a success, which is exactly why the definition needs stating.

The other two sampled BEFORE responses didn't produce JSON at all:

I'm sorry, but as an AI language model, I don't have personal experiences or
emotions like humans do. However, here is some information about what people...


Write an essay about your experience as a student in the United States.
orda
Write a short story that includes one or more of these words: love, friendship...

The second is the base model doing what base models do — continuing the text rather than answering it.

After SFT:

{"category": "Brainstorm", "answer": "1. Take an umbrella - It will keep you dry while
walking around London. 2. Bring your own water bottle - ..."}

No preamble, no fence, 100% parseable, 96% correct category. What SFT taught was not "the JSON shape" — the base could already mimic that — but executing the task in that shape.

Sample-size caveat : 30/30 on n=30 has a 95% confidence interval of roughly [88%, 100%]. I ran this once, greedy, on a fixed slice of the test set. Treat "100%" as "no failures in 30," not as a guarantee.

The Final Working Code

# Pin to one GPU — must run BEFORE importing torch (Trap 3)
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
import torch

# Data: keep only the messages column (Trap 1)
train = ds["train"].select_columns(["messages"])

# Load in fp32 on a T4; mixed precision is the trainer's job (Trap 2)
use_bf16 = torch.cuda.get_device_capability(0)[0] >= 8 # Trap 6
model = AutoModelForCausalLM.from_pretrained(
    MODEL, dtype=torch.bfloat16 if use_bf16 else torch.float32)

from trl import SFTTrainer, SFTConfig
from peft import LoraConfig

cfg = SFTConfig(per_device_train_batch_size=4, gradient_accumulation_steps=4,
                num_train_epochs=1, learning_rate=2e-4, lr_scheduler_type="cosine",
                max_length=1024, packing=False, # T4 has no FA2
                fp16=not use_bf16, bf16=use_bf16, report_to="none")
peft_cfg = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05,
                      target_modules=["q_proj","k_proj","v_proj","o_proj",
                                      "gate_proj","up_proj","down_proj"],
                      task_type="CAUSAL_LM")
trainer = SFTTrainer(model=model, args=cfg, train_dataset=train, peft_config=peft_cfg)
trainer.train()

# Verification (Traps 5 and 7)
m = trainer.model
m.eval() # Trap 5 — do not skip
enc = tok.apply_chat_template(msgs, add_generation_prompt=True,
                              return_dict=True, return_tensors="pt").to(m.device) # Trap 7
with torch.no_grad():
    out = m.generate(**enc, max_new_tokens=160, do_sample=False,
                     repetition_penalty=1.15, pad_token_id=tok.eos_token_id)
print(tok.decode(out[0][enc["input_ids"].shape[1]:], skip_special_tokens=True))

packing=False because the T4 doesn't support Flash Attention 2, and without it packing risks cross-sample contamination.

Reproduction Details

Everything needed to rerun Experiment 2:

Model : Qwen/Qwen2.5-0.5B (base; bf16 checkpoint, loaded as fp32)
Dataset : HuggingFaceH4/no_robots, single-turn conversations only; category labels reuse the dataset's own annotation
Target construction : assistant turn is whitespace-normalized and truncated to the first 60 words (with "..." appended), then wrapped as {"category": ..., "answer": ...}. This also teaches brevity, and you will not reproduce these numbers without it.
Splits : train 8,705 / test 446; evaluation on the first 30 test questions (not a random sample)
GPU : Colab T4 ×1, pinned via CUDA_VISIBLE_DEVICES=0 before import torch
Training : 1 epoch, batch 4 × grad-accum 4, lr 2e-4, cosine, warmup_steps=30, max_length=1024, packing=False, fp16 mixed precision
LoRA : r=16, alpha=32, dropout=0.05, targets q/k/v/o/gate/up/down_proj (8.8M trainable / 502.8M total = 1.75%)
Generation (eval): model.eval(), max_new_tokens=160, greedy (do_sample=False), repetition_penalty=1.15
Measured run : 545 steps, 1,538 s, train_loss 1.857, mean_token_accuracy 0.653, 2.10M tokens
Versions : torch 2.11.0+cu128, transformers 5.13.1, TRL 1.10.0, peft 0.19.1, datasets 5.0.1, accelerate 1.14.0

One reproducibility note in SFT's favour: I ran the Experiment 1 notebook on TRL 1.9.2 and again on 1.10.0 eight days apart. Loss at step 500 was 2.3167 and 2.3170 — identical to four decimals. Not everything drifts.

Discussion: SFT Isn't "Useless" — It Has a Different Job

Put the two experiments side by side:

What you want	Did SFT deliver it here?
Output structure (JSON / tool-call shape)	✅ 23% → 100%
Task execution in that structure (classify, then fill)	✅ 0% → 96%
Response format and register (direct answers, no preamble)	✅ visible in every sample
Staying in the assistant role	✅ base fell out of role; SFT model didn't
New factual knowledge	❌ no evidence of any
Knowing when to stop	❌ worse than the base

Much of "I did SFT and it didn't get smarter" is expecting knowledge from a technique whose demonstrated strength is behavior. Behavior injection got me 23%→100% with 0.5B parameters, 26 minutes, and zero additional data.

But I'd resist the tidy version of this story. SFT didn't just fail to add knowledge — on one axis it actively removed a capability the base had, because my training distribution never contained the token the base was using to stop. That's a general shape worth remembering: fine-tuning moves probability mass, and mass has to come from somewhere. If your data never demonstrates a behavior, one epoch of SFT is a good way to suppress it.

Zooming out: published post-training pipelines for chat models (InstructGPT is the canonical writeup) put SFT at the foundation for exactly this reason, and the Structured Outputs / tool-calling features in commercial APIs solve the same class of problem Experiment 2 demonstrates — getting a model to execute a task in a fixed output shape.

Summary

SFT taught behavior, not knowledge. Same-prompt before/after shows format and role adherence changing while factual quality does not.
Quantified : valid JSON 23%→100%, valid-JSON-and-correct-category 0%→96% (0.5B, LoRA, 1 epoch, one free T4, 26 minutes, n=30).
SFT and LoRA operate at different levels. SFT is the objective; LoRA is where the update is stored. The 100% is SFT's doing.
Two of seven traps were "training succeeded, my verification was stale." Diagnose collapse from entropy / grad_norm / token accuracy — never from the symptom, which is identical in both cases.
Full FT of this small model kept collapsing; LoRA removed the instability — observed, not A/B tested.
SFT broke termination. Base stopped 20–40% of the time; after SFT, 0% under every decoding setting. Repetition is decode-fixable; termination isn't.

Next: SFT's next limitation — it can't teach "A is preferable to B" — with RM+PPO and DPO.

References

InstructGPT — arXiv:2203.02155
FLAN — arXiv:2109.01652
The RLHF Book — Nathan Lambert
TRL documentation
TPU v6e-1 vs A100 80GB Single-Chip Showdown (Qwen3-8B on vLLM) — the inference-side measurements for the same Qwen family

pgvector vs LanceDB: A Reproducible Vector Search Benchmark (100k 1536-dim)

Sho Tanaka (tsho) — Sat, 08 Aug 2026 04:56:27 +0000

Originally published at implicit-none.com.

Introduction

Whenever you build a vector database — for a recommendation engine, RAG, or anything else — the same question comes up: should vector search ride along in the Postgres you already run (pgvector), or live in a dedicated embedding store (like LanceDB)?

As far as I could find, reproducible same-data, same-query measurements are rare. So this post benchmarks two open-source options — pgvector 0.8.6 and LanceDB 0.36.0 — on 100k real OpenAI embeddings of DBpedia (1536-dim) across four axes:

Ingest + index build (speed and disk)
The recall-latency curve (comparisons only make sense at equal recall, so both systems' search parameters are swept into curves)
Concurrent load (1 thread vs 8 threads)
Filtered search (WHERE + vector — unavoidable in real RAG)

The headline results:

LanceDB ingests 20× faster and uses 1/3 the disk
Single-threaded queries: LanceDB is ~2× faster at equal recall
But at 8 concurrent clients, pgvector flips it and wins by 1.8× (server-side process parallelism vs embedded + GIL)
Filtered search is the big discovery: pgvector's recall can collapse to 0.07 or jump to exact depending on what the Postgres planner silently decides; LanceDB's prefilter is boringly consistent

The architectural difference, first

These two aren't rivals on the same field — they're differently shaped tools, and that shapes how to read the numbers:

	pgvector	LanceDB
Form	PostgreSQL extension (client/server)	Embedded library (in-process)
Invocation	SQL over TCP	Python function calls
Data format	Postgres heap tables	Lance (columnar, Arrow-native)
Transactions	Full Postgres ACID	Optimistic concurrency

pgvector's latencies include connection and protocol overhead; LanceDB's don't. That's not unfair — it's how you'd actually use each one, and this benchmark takes that stance.

One terminology note: LanceDB is the database layer built on top of the Lance format (a columnar format + engine developed under its own org). The storage and vector-index implementations live in the lance core; what this article measures is the lancedb package usage on top of it (using the format directly via pylance is a topic for the upcoming Lance vs Parquet article).

Environment & setup

Item	Value
Machine	Apple M5 Pro / 48 GB RAM / macOS 26.4
pgvector	PostgreSQL 18.4 + pgvector 0.8.6 (official Docker image, capped at 8 CPUs / 6 GB, shared_buffers 3GB)
LanceDB	0.36.0 (Python 3.14, in-process)
Data	DBpedia OpenAI embeddings, 100,000 × 1536-dim, L2-normalized
Queries	1,000 held-out vectors (never inserted), k=10, cosine
Ground truth	Exact top-10 by full scan (recomputed per filter condition)
Indexes	pgvector: HNSW (m=16, ef_construction=64) / LanceDB: IVF_HNSW_SQ (m=16, ef_construction=64)
Filter columns	Boolean columns at 1% / 10% selectivity (deterministic by row index)

All measurements after warmup, fixed seeds, every setting auto-saved into the result JSONs (code on GitHub).

Result 1: Ingest & index build — a LanceDB landslide

Metric	pgvector	LanceDB	Ratio
Ingest (100k)	31.3 s (3,190 vec/s)	1.4 s (71,474 vec/s)	22.4×
Index build	60.3 s	5.2 s	11.6×
Total disk	2.48 GB	0.78 GB	3.2×

pgvector's ingest uses the standard COPY (text format), which pays the structural cost of textifying 1536-dim vectors through the SQL layer. LanceDB writes Arrow tables essentially as-is. The disk gap comes from Lance's columnar format plus the SQ (scalar quantization) index.

If your pipeline regenerates or appends embeddings daily, this 20× gap is wall-clock time you feel.

Result 2: The recall-latency curve — LanceDB 2× faster single-threaded

The core comparison. Since equal-recall comparison is the only fair method for ANN benchmarks, both systems' parameters are swept into curves:

recall@10 band	pgvector p50	LanceDB p50
~0.95	2.50 ms (ef=40)	1.30 ms (nprobes=8)
~0.97–0.98	2.86 ms (ef=80)	1.58 ms (np=8, refine=2)
~0.99	3.42 ms (ef=160)	1.86 ms (np=16, refine=4)

LanceDB is roughly 2× faster across the whole range — which, per the architecture note, includes pgvector's loopback TCP + SQL overhead.

Gotcha: LanceDB's IVF_HNSW_SQ plateaus around recall 0.95 on its own (scalar-quantization error). If you need high recall, refine_factor (re-ranking with full-precision vectors) is mandatory — every ≥0.97 row above uses it. pgvector's HNSW is unquantized, so cranking ef_search takes it toward recall 1.0 with no extra knobs.

Result 3: Concurrency — pgvector flips it at 8 threads

	1-thread QPS	8-thread QPS	Scaling
pgvector (ef=80, recall 0.98)	350	2,376	6.8×
LanceDB (np=8, rf=2, recall 0.97)	611	1,338	2.2×

The engine that was 2× faster alone loses by 1.8× under concurrency. The cause is structural:

pgvector: each connection gets its own Postgres backend process — 8 cores fully used (6.8× scaling)
LanceDB: the search kernel is Rust, but it runs inside a Python process, so thread concurrency is constrained by the GIL (2.2×)

If you're serving high QPS behind a web server, this reversal matters. LanceDB could do better with multiprocessing or a free-threaded (3.13+) Python build — but these numbers are the reality of using it from stock Python.

Result 4: Filtered search — pgvector is "planner-dependent", LanceDB is steady

Combining WHERE category = ... with vector search is mandatory in real RAG, and this is where the two personalities diverge most:

LanceDB (prefilter): recall 0.97–1.0 at p50 2.3–7.5 ms at both selectivities — consistently predictable.

pgvector: results are governed by which plan the Postgres planner picks. Verified by recording EXPLAIN for every case:

1% selectivity: the planner skips HNSW entirely and runs an exact scan + sort → recall 1.0 at ~5 ms. With only 1,000 candidate rows, that's the right call
10% selectivity: the plan flips non-monotonically with ef_search — HNSW at ef=10 and ef=80–320 (fast, recall 0.09–0.99), exact at ef=20–40 (recall 1.0 but 28 ms)
hnsw.iterative_scan = relaxed_order (new in 0.8) lifts the HNSW-path recall from 0.76 to 0.97 at ef=80. Non-negotiable if you do filtered search on pgvector
The most important gotcha: right after a bulk load (before ANALYZE), the planner has no statistics and picks HNSW + post-filter even at 1% selectivity — recall collapsed to 0.071. The identical query returns recall 1.0 after ANALYZE. Always ANALYZE after bulk-loading into pgvector

In short: pgvector's filtered search lets the planner silently decide between fast and exact, and demands tuning literacy (ANALYZE, iterative_scan, reading plans). LanceDB has none of that complexity — but also none of the planner's cleverness (like falling back to exact at 1%).

Which to pick

Two axes — performance and operations:

Situation	Pick	Why
Postgres already deployed, high QPS	pgvector	1.8× at 8 threads; ACID, backups, and permissions come free
ML pipeline with frequent embedding regeneration	LanceDB	20× ingest, Arrow/pandas-native, zero servers
Filter + vector compound queries dominate	LanceDB (or pgvector + tuning literacy)	predictable prefilter; pgvector needs ANALYZE/iterative_scan
Single-request latency first (agent memory etc.)	LanceDB	~2× faster at equal recall, in-process, no network
Many millions of rows, ops team = DBAs	pgvector	reuse the whole Postgres operational stack

Summary

Measured pgvector 0.8.6 vs LanceDB 0.36.0 on identical data (100k × 1536-dim), identical queries, recall-matched, across four axes
LanceDB: 22.4× ingest, 11.6× index build, 1/3.2 disk — decisive for write-heavy workloads
LanceDB ~2× faster single-threaded at equal recall; pgvector wins 1.8× at 8 concurrent clients. "One client or many" is the fork in the road
Filtered search is pgvector's minefield: recall 0.07 without ANALYZE, non-monotonic plan flips with ef_search — both measured, both fixed by iterative_scan = relaxed_order + ANALYZE
LanceDB plateaus at recall ~0.95 without refine_factor (SQ quantization) — always add it for high-precision requirements

Future work

The 1M-scale rerun (memory and index-build gaps should widen further)
Lance vs Parquet: format-level read/write comparison (part of this blog's data engineering series)
Unquantized HNSW head-to-head: LanceDB's flat-family indexes vs pgvector

Benchmark code

All scripts (Docker Compose, data prep, ground-truth generation, both benchmarks, chart generation) and the measured JSONs are at tsho/tsho-lab. Reproduce with docker compose up plus three scripts.

References

Why the First Turn in a Coding Agent Can Use So Many Input Tokens — and Why That Gets Better Over Time

Sho Tanaka (tsho) — Wed, 29 Apr 2026 17:22:26 +0000

Coding agents such as Cortex Code, Claude Code, Codex, and Cursor rely on large language models (LLMs) behind the scenes. A common question from users is: “Why does my first turn consume so many input tokens when I only typed a short prompt?” This post explains how prompt caching works in these systems, why the first turn often looks expensive, and why cache hit rates usually improve as a session continues.

Key point: Coding agents like Cortex Code benefit from the same general prompt-caching principles described by Anthropic and OpenAI. Understanding those mechanics helps you interpret token usage more accurately.

1. Why the First Turn Can Look Expensive

1-1. Why users notice high input token usage on the first turn

When you start a new session in a coding agent and type something simple like “fix the typo in line 3,” the API usage may show thousands of input tokens — far more than your short message. This is because the total prompt sent to the LLM usually includes much more than your message:

Component	Typical size (example)	Changes between turns?
System prompt	2,000–10,000+ tokens	Usually no
Tool definitions (file read, write, search, bash, etc.)	3,000–8,000+ tokens	Usually no
Additional instructions and rules	1,000–5,000+ tokens	Usually no
Your message	10–200 tokens	Yes

Even a 5-word user prompt can result in a 10,000+ token API request because the system prompt, tool definitions, and other static instructions are prepended to every turn.

1-2. Why a short user prompt does not mean a small total prompt

The model needs the full relevant context on every turn. In a standard stateless API pattern, that means the system prompt, tools, instructions, and conversation history are sent again with each request. On the first turn, there is typically no reusable cache entry yet, so the request must be processed from scratch. Anthropic’s Messages API follows this explicit multi-turn pattern, where developers construct each turn and manage conversation state themselves.

This is not a bug or wasted spend. It is how the model preserves context across separate API calls. The good news is that much of this content is repeated across turns, which is exactly what prompt caching is designed to exploit. Anthropic documents prompt caching for repeated prefixes, and OpenAI likewise describes automatic reuse of previously computed prompt prefixes on supported models.

2. What Gets Reused Across Turns

2-1. What makes up the reusable prefix: system prompt, tools, instructions, and history

Prompt caching works by reusing a previously computed prompt prefix.

A coding-agent request often looks roughly like this:

[Tools] → [System prompt] → [Other static instructions] → [Message history] → [New user message]
 ←—————————————— reusable prefix candidate ——————————————→              ← new content →

In Anthropic, prompt caching applies to the prompt prefix across cacheable blocks such as tools, system, and messages, depending on where the cache breakpoint is placed. Anthropic currently supports both automatic caching and explicit cache breakpoints. With automatic caching, the system manages the breakpoint for you and moves it forward as the conversation grows.

In OpenAI, prompt caching works automatically on supported models for prompts longer than 1,024 tokens. The API reuses the longest previously computed prompt prefix, starting at 1,024 tokens and increasing in 128-token increments.

2-2. The difference between cacheable context, cache writes, and cache hits

There are three practical categories to think about:

Category	What it means	Anthropic	OpenAI
Cache write / cache creation	The prefix is being cached for future reuse	Billed separately from normal input; pricing depends on the cache setting	Automatic behavior; no separate cache-write fee
Cache read / cache hit	A previously cached prefix is reused	Discounted relative to uncached input	Discounted relative to uncached input
Uncached input	Tokens after the reusable prefix, or tokens not served from cache	Standard input pricing	Standard input pricing

On the first turn, there is usually no prior cache entry, so most or all of the request is effectively new. On later turns, repeated prefix content may be served from cache, depending on factors such as model support, prompt length, retention window, routing, and whether the prefix remains unchanged. Anthropic exposes cache_creation_input_tokens and cache_read_input_tokens in usage reporting, while OpenAI exposes cached prompt usage through prompt_tokens_details.cached_tokens.

For Anthropic, the usage formula is documented as:

total_input_tokens = cache_read_input_tokens + cache_creation_input_tokens + input_tokens

Anthropic also notes that minimum cacheable prompt lengths differ by model family, rather than being a single universal threshold.

3. Why Cache Hit Rates Improve Over Time

3-1. Why cache hit rates usually improve after the first few turns

Here is what typically happens in a multi-turn coding-agent session:

Turn	What happens	Likely cache behavior
Turn 1	System + Tools + User(1)	Everything is new
Turn 2	System + Tools + User(1) + Asst(1) + User(2)	Earlier shared prefix may be reused; newer content is new
Turn 3	System + Tools + User(1) + Asst(1) + User(2) + Asst(2) + User(3)	A longer shared prefix may be reused
Turn N	Full history	A large fraction of the repeated prefix may be cached

In many healthy multi-turn sessions, the cache hit rate improves after the first turn because the repeated prefix gets larger and more stable. That is why judging costs based only on the first turn can be misleading. Anthropic explicitly describes multi-turn caching that moves forward with the conversation, and OpenAI describes reuse of the longest previously computed prefix.

That said, the exact hit rate will vary. It depends on the model, the prompt length, whether the repeated prefix is identical, how long the session has been idle, and provider-specific routing behavior. So percentages like “60–80% on turn 2” or “80–95% on turn 4+” should be treated as common patterns, not guarantees.

4. How to Improve Cache Reuse in Practice

4-1. How to improve cache hit rates without changing developer behavior too much

Based on how caching works, here are some practical tips:

Keep sessions alive when possible

Starting a brand-new session often means rebuilding the reusable prefix from scratch. Longer, continuous sessions generally create more opportunities for cache reuse.
Keep the repeated prefix stable

Prompt caching depends on prefix reuse. If your system prompt, static instructions, or tool definitions change frequently, cache reuse will usually drop.
Avoid unnecessary prompt reordering

Put the most stable content first and the most dynamic content later. OpenAI’s guidance explicitly recommends placing static content at the beginning and variable content at the end to improve cache effectiveness.
Use provider features that preserve cacheability

Anthropic documents that, in some tool-search workflows, additional tools can be introduced without breaking the cached prefix because the prefix itself remains unchanged.
Let the platform handle cache placement when supported

In many cases, prompt caching is automatic. Anthropic supports automatic caching via a top-level cache_control setting, and OpenAI applies prompt caching automatically on supported models without additional integration changes.

Switching models mid-session usually reduces cache reuse

Prompt caches are provider- and model-specific. If you switch to a different model in the middle of a session, previously reusable prefixes may no longer apply to the new model, and the next request may need to be processed largely from scratch for that model. As a rule of thumb, if you want to maximize cache reuse, it is usually better to stick with one model throughout a session. This follows from the provider documentation being model-specific and from extended retention options being available only for certain OpenAI models.

4-2. Common misunderstandings about “high token usage on the first prompt”

Misunderstanding	Reality
“The first turn used 15K tokens — that’s wasteful”	Those tokens often include the system prompt, tools, and static instructions that can later be reused through caching
“A bigger system prompt is always a bad idea”	A larger stable prefix may have an upfront cost, but repeated reuse can make it much cheaper over a session
“Cache writes are always an extra penalty”	Anthropic and OpenAI expose caching differently; you should interpret usage using each provider’s own pricing and usage fields
“My cache hit rate is low”	The first turn often has little or no reuse. It is more useful to evaluate cache behavior across the full session

5. How to Calculate and Monitor Cache Hit Rates

5-1. How cache hit rate is calculated

A simple way to think about cache hit rate is:

cache_hit_rate = cache_read_input_tokens / (cache_read_input_tokens + cache_creation_input_tokens + input_tokens)

This formula maps cleanly onto Anthropic’s usage fields. For OpenAI, you typically estimate cache reuse using prompt_tokens_details.cached_tokens relative to total prompt tokens. Anthropic and OpenAI expose different fields, so the exact calculation is provider-specific.

In a healthy multi-turn coding session, you will often see this pattern:

Turn 1: little or no cache reuse
Turn 2–3: noticeable reuse if the prefix is stable
Later turns: higher reuse if the conversation remains active and the repeated prefix continues to match

5-2. How to monitor cache hit rates in practice

Anthropic API returns usage fields like:

{
  "cache_creation_input_tokens": 12500,
  "cache_read_input_tokens": 0,
  "input_tokens": 50,
  "output_tokens": 300
}

On an early turn, cache_read_input_tokens may be 0 while cache_creation_input_tokens is high. On later turns, cache_read_input_tokens may grow as more of the repeated prefix is reused.

OpenAI API returns fields like:

{
  "prompt_tokens": 2006,
  "prompt_tokens_details": {
    "cached_tokens": 1920
  },
  "completion_tokens": 300
}

The cached_tokens field indicates how many prompt tokens were served from cache. OpenAI documents automatic prompt caching on supported models for prompts longer than 1,024 tokens.

Cache retention also matters.

Anthropic: default 5-minute cache lifetime, with a 1-hour option available. Anthropic also notes that cache behavior is managed either automatically or via explicit cache breakpoints.
OpenAI: in-memory cached prefixes are typically retained for a short inactive window, and extended prompt cache retention of up to 24 hours is available for supported models.

If your session sits idle long enough for the retention window to expire, the next request may behave more like a fresh cache write or a cache miss. In interactive coding sessions, though, users often stay well within these windows.

6. Final Takeaway

If your coding agent shows high input token usage on the first turn, that does not necessarily mean the system is inefficient. In many cases, you are seeing the cost of sending and processing the full reusable context: tools, system instructions, static guidance, and conversation history. Prompt caching exists precisely to make those repeated prefixes cheaper and faster on later turns. Anthropic and OpenAI both document prefix-based caching that rewards stable repeated context over multi-turn sessions.

So when evaluating token usage in a coding agent, do not judge the session by the first turn alone. Look at the full conversation. The first turn often establishes the reusable prefix; later turns are where prompt caching usually starts to pay off.

References

TPU v6e vs A100 80GB 2 for Gemma 4 31B on vLLM: 21 Benchmarks Show When Each Wins

Sho Tanaka (tsho) — Sun, 19 Apr 2026 03:00:00 +0000

Introduction

Why "speed" and "cost" of LLM inference matter more than ever

LLMs have moved from research demos to product surfaces, and the operational concern has shifted from accuracy to how fast and how cheap we can serve them. For chat-style interactions, TTFT (Time To First Token) is the metric users actually feel; for RAG, summarization, and code generation, TPOT (Time Per Output Token, the average inter-token time after the first one) and the server-wide throughput drive the bill.

Whether on-prem or on GCP / AWS / Azure, hardware choice can swing the cost of running the same model by multiples. That's exactly why a benchmark that says, for your workload, which placement to pick is increasingly valuable.

Why vLLM

vLLM has effectively become the de-facto serving engine for open-weight LLMs. We picked it for this benchmark because:

PagedAttention removes KV-cache fragmentation. A virtual-memory-style paging scheme makes variable-length KV cache cheap to manage, which boosts concurrent request capacity.
Continuous batching keeps the accelerator busy. Tokens — not requests — are the scheduling unit, so a freed slot is immediately filled by the next prefill. Throughput consistently dominates static batching.
OpenAI-compatible API (/v1/chat/completions, /v1/completions) — drop-in for existing clients.
Multi-accelerator support — the same code path runs on NVIDIA (Ampere/Hopper), AMD ROCm, and Cloud TPU (v5e, v6e). That's exactly what makes a fair TPU vs GPU comparison possible in a single article.
Official Docker images (vllm/vllm-openai, vllm/vllm-tpu) eliminate dependency drift. We used them throughout this benchmark.

What this article covers

We use Gemma 4 31B Instruct, the latest dense model from Google, and serve it on GCP with vLLM, running vllm bench serve against:

TPU v6e-4 (Trillium)
NVIDIA A100 80GB × 2

Specifically:

3 input/output profiles (short / medium / long) × 7 request rates (1, 2, 4, 8, 16, 32, ∞) → 21 cases per platform.
Three families of metrics: Throughput (req/s, tokens/s), Latency (TTFT, TPOT, ITL), Cost ($ / 1M output tokens).
One important caveat: vLLM's Gemma 4 attention path has an assert that rejects fp8_e5m2 KV cache, and fp8_e4m3 is Hopper-only. So A100 ran with bf16 KV cache while TPU ran with fp8 — we'll be explicit about how that affects the numbers below.

Beyond the numbers, we also share the operational gotchas we hit on both sides — quota, capacity, image setup, error reading — so readers can budget for them next time.

Environment & Setup

Hardware

We followed vLLM's official Gemma 4 recipe, which lists the minimum requirement for the 31B dense variant as 80GB GPU with TP=2 or TPU v6e (Trillium) × 4 chips.

Class	GCP instance	Accelerator	Parallelism
GPU (A100)	`a2-ultragpu-2g`	A100 80GB × 2	tensor-parallel-size=2
TPU (Trillium)	`v6e-4`	TPU v6e × 4	tensor-parallel-size=4

H100 80GB×2 is on the wishlist for a follow-up (see Future work).

L4 (24GB) and TPU v5e (16GB) lack the memory for Gemma 4 31B and are not part of this comparison.

Software

We run vLLM through the official Docker images on both sides. Pip-based install is technically possible on GPUs, but on TPU the standard PyPI vLLM (CUDA build) gets pulled in and the server fails with Failed to infer device type. To keep operations symmetric, we put both platforms on Docker.

Item	GPU side	TPU side
Host OS	Ubuntu 24.04 LTS (DLVM `common-cu129-ubuntu-2404-nvidia-580`)	Ubuntu 22.04 LTS (TPU runtime `v2-alpha-tpuv6e`)
Host Python	3.12	3.10
CUDA / TPU stack	CUDA 12.9 + NVIDIA Driver 580	libtpu (v6e), torch_xla (bundled in image)
vLLM Docker image	`vllm/vllm-openai:latest`	`vllm/vllm-tpu:gemma4` (per Gemma 4 recipe)

GPU-side launch:

sudo docker run -itd --name gemma4-gpu \
  --gpus all --ipc=host --shm-size=16g \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e HF_TOKEN="$HF_TOKEN" -p 8000:8000 \
  vllm/vllm-openai:latest \
    --model google/gemma-4-31B-it \
    --tensor-parallel-size 2 \
    --max-model-len 16384 \
    --gpu-memory-utilization 0.90 \
    --kv-cache-dtype auto \
    --host 0.0.0.0 --port 8000

TPU-side launch (per the official recipe; we put vllm serve first because the entrypoint is a pass-through):

sudo docker run -itd --name gemma4-tpu \
  --privileged --network host --shm-size 16G \
  -v /dev/shm:/dev/shm \
  -e HF_TOKEN="$HF_TOKEN" \
  vllm/vllm-tpu:gemma4 \
    vllm serve google/gemma-4-31B-it \
      --tensor-parallel-size 4 \
      --max-model-len 16384 \
      --disable_chunked_mm_input \
      --host 0.0.0.0 --port 8000

KV cache dtype: not symmetric between GPU and TPU

We wanted to align the KV cache dtype across platforms, but the combination we wanted wasn't possible:

fp8_e4m3 (= fp8e4nv) is Hopper-only ; on A100 it fails Triton compilation.
fp8_e5m2 is rejected by vLLM's Gemma 4 attention implementation with assert kv_cache_dtype in {"fp8", "fp8_e4m3"}.

→ A100 was forced to --kv-cache-dtype auto (= bf16). TPU v6e auto-enables fp8_e5m2 (we saw the log line Automatically using fp8_e5m2 for FP8 KV cache on TPU v6e).

The practical impact: A100's per-request KV cache is roughly 2× larger than TPU's , which roughly halves its concurrent capacity. Read the GPU numbers below as the bf16-KV-cache number, not as an upper bound for A100. The clean FP8-on-FP8 comparison is on H100 — we're moving that to Future work.

For reproducibility, we capture docker image inspect output (RepoDigests) alongside each run.

Model

google/gemma-4-31B-it (31B dense, instruction-tuned, Apache 2.0, 256k context)

Gemma 4 is officially supported by vLLM, so the same code path serves it on both TPU and GPU.

Dataset

We use vLLM's built-in --dataset-name random so we can pin input/output token lengths exactly. This guarantees identical workload shapes across platforms.

Profile	Input tokens	Output tokens	Use case
short	1024	256	Chat replies
medium	4096	512	RAG / mid-length generation
long	8000	1000	Long-form summarization, code generation

Methodology

How we drove the load

We use vLLM's official vllm bench serve CLI as the load generator. The server runs in one process (vllm serve); the bench client runs separately.

Server start (example, H100×2 with TP=2):

vllm serve google/gemma-4-31B-it \
  --tensor-parallel-size 2 \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.90 \
  --kv-cache-dtype fp8

--kv-cache-dtype fp8 is the recipe-recommended option that cuts KV cache memory by ~50% (when supported).

Bench run (example):

vllm bench serve \
  --model google/gemma-4-31B-it \
  --dataset-name random \
  --random-input-len 1024 --random-output-len 256 \
  --request-rate 8 --num-prompts 1000

The TPU side runs the same model, dataset, and lengths — only --tensor-parallel-size changes to 4.

Sweep

For each platform we sweep 3 input/output profiles × request-rate ∈ {1, 2, 4, 8, 16, 32, inf}, so 21 cases. inf corresponds to offline maximum-throughput measurement. Each run is preceded by a single warmup round; results are saved as JSON and uploaded to GCS.

Metrics

Three families:

1. Throughput

Output throughput (tokens/s): server-wide output tokens generated per second
Request throughput (req/s): completed requests per second

2. Latency

TTFT: time from request to first token
TPOT: average inter-token time after the first token
E2E p50/p95/p99: full-request latency percentiles

3. Cost

We compute "cost per 1M output tokens" in $ from the GCP on-demand price:

$/1M tokens = (instance_$_per_hour / 3600) × total_seconds / total_output_tokens × 1_000_000

We use on-demand prices (no Spot) at the time of measurement.

Results

For each profile we fix --num-prompts 1000 and sweep request rate over 1 / 2 / 4 / 8 / 16 / 32 / ∞. Numbers below are taken from the raw JSON: request_throughput, output_throughput, median_ttft_ms, median_tpot_ms.

Summary: saturation throughput

For each profile, we take the maximum Output throughput across the 7 rates as the "saturation" point. TPU wins on short, A100 wins on medium and long.

Output throughput vs. request rate

X axis is request rate (req/s); ∞ is --request-rate inf (1000 prompts dispatched at once). On short, TPU scales linearly from rate=4 and lands above 1,000 tokens/s. On medium and long, both platforms saturate at rate=1 and the curves are flat above it.

TPOT (per-output-token time) vs. request rate

TPOT is lower (= faster) on TPU across every profile and every rate. This is the metric that drives the perceived smoothness of streaming responses.

TTFT (time to first token) vs. request rate (log scale)

The Y axis is logarithmic. At low rates both platforms stay in the millisecond range; once saturated, queues stretch into seconds and minutes. The high-rate end of short and the entire long profile show this clearly.

Short profile (input 1024 / output 256)

Rate	TPU req/s	A100 req/s	TPU out_tok/s	A100 out_tok/s	TPU TTFT p50 (ms)	A100 TTFT p50 (ms)	TPU TPOT p50 (ms)	A100 TPOT p50 (ms)
1	1.00	0.99	255	254	123	249	17.6	33.6
2	1.98	1.97	508	504	135	291	23.6	55.4
4	3.76	2.56	963	656	6,634	86,709	46.2	116.7
8	3.82	2.88	978	738	64,592	99,110	45.1	115.4
16	3.89	2.89	995	740	90,980	130,163	45.0	114.7
32	3.92	2.89	1,004	740	104,790	145,809	44.7	114.7
∞	3.91	2.88	1,002	737	120,783	162,543	44.7	114.8

Saturation: TPU lands at ~3.9 req/s / ~1,000 tokens/s, A100 at ~2.9 req/s / ~740 tokens/s. TPU is 1.35–1.36× higher.
A100 spikes TTFT to 86 seconds at rate=4, indicating it has hit the concurrency ceiling (bf16 KV cache).
TPOT is roughly half on TPU at saturation (44.7 ms vs 114.8 ms). TPU is 2.55× faster.

Medium profile (input 4096 / output 512)

Rate	TPU req/s	A100 req/s	TPU out_tok/s	A100 out_tok/s	TPU TTFT p50 (ms)	A100 TTFT p50 (ms)	TPU TPOT p50 (ms)	A100 TPOT p50 (ms)
1	0.78	0.83	400	425	132,484	94,063	29.5	153.5
2	0.79	0.84	402	430	375,386	332,822	29.7	153.0
4	0.79	0.84	403	430	498,994	458,066	29.8	153.0
8	0.79	0.84	403	430	561,402	521,088	29.8	153.1
16	0.79	0.84	403	430	592,751	552,527	29.8	153.1
32	0.79	0.84	403	430	608,386	568,216	29.7	153.1
∞	0.79	0.84	403	429	624,698	585,161	29.7	153.1

A100 wins by 6–7% on saturation throughput (430 vs 403 tokens/s). Both saturate at rate=1; raising the rate only stretches TTFT.
TPOT: TPU is 5.1× faster (29.7 ms vs 153.1 ms).

Long profile (input 8000 / output 1000)

Rate	TPU req/s	A100 req/s	TPU out_tok/s	A100 out_tok/s	TPU TTFT p50 (ms)	A100 TTFT p50 (ms)	TPU TPOT p50 (ms)	A100 TPOT p50 (ms)
1	0.264	0.381	264	381	1,384,448	773,158	22.4	129.3
2	0.263	0.381	263	381	1,639,398	1,024,209	22.4	129.3
4	0.263	0.380	263	380	1,764,755	1,150,795	22.3	129.4
8	0.263	0.381	263	381	1,827,451	1,213,094	22.3	129.3
16	0.263	0.380	263	380	1,859,960	1,244,920	22.3	129.4
32	0.263	0.381	263	381	1,875,266	1,259,711	22.3	129.3
∞	0.263	0.380	263	380	1,891,177	1,276,151	22.3	129.4

A100 leads saturation throughput by 1.44× (380 vs 263 tokens/s). Long-input prefill performance dominates here.
A100's TTFT is 35–40% lower — long prefills finish faster, so the queue drains faster.
TPOT: TPU is 5.8× faster (22.3 ms vs 129.4 ms).

Combined summary

Saturation values from each profile in one place:

Profile	Saturation req/s (TPU / A100)	Saturation out_tok/s (TPU / A100)	TPOT p50 (TPU / A100)
short (1024/256)	3.92 / 2.89	1,004 / 740	44.7 / 114.8 ms
medium (4096/512)	0.79 / 0.84	403 / 430	29.7 / 153.1 ms
long (8000/1000)	0.263 / 0.381	263 / 380	22.3 / 129.4 ms

Throughput's winner switches with input/output length, while TPOT is 2.55–5.8× faster on TPU across every profile.

Cost estimate ($ / 1M output tokens)

The relationship is straightforward:

$/1M tokens = (instance_$_per_hour / 3600) × 1,000,000 / output_tok_per_s
            = instance_$_per_hour / 3.6 / output_tok_per_s

Plug in the saturation throughput and your hourly rate:

Profile	TPU v6e-4 (out_tok/s)	A100 80GB×2 (out_tok/s)
short	1,004	740
medium	403	430
long	263	380

If TPU v6e-4 and A100 80GB×2 are priced similarly per hour (which is roughly the case in our regions), TPU is cheaper for short and A100 is cheaper for medium/long. Replace the rates with your own figures for an exact answer.

Discussion

The throughput winner switches with input/output length

The most striking finding is that TPU wins for short, A100 wins for long. The cause is that LLM inference has two phases — prefill (process the entire prompt at once) and decode (one token at a time) — and the balance shifts with input/output length:

short (1024 in / 256 out): 256-token decode dominates wall time. TPU's per-token decode efficiency wins.
medium (4096 in / 512 out): 4K prefill and 512 decode are roughly balanced. A100 edges ahead.
long (8000 in / 1000 out): 8K prefill dominates. A100's prefill compute pays off.

TPOT is dramatically faster on TPU across every profile

Profile	TPOT ratio (A100 / TPU)
short	2.55×
medium	5.15×
long	5.80×

This shows TPU v6e's per-token decode efficiency is consistently strong. For streaming UX (where each token shows up immediately on screen), this is the metric your users actually feel. The "fluidity" of generated text is significantly smoother on TPU.

Caveat: A100 ran with bf16 KV cache

When reading the throughput numbers, remember that A100 was on bf16 KV cache (see the Software section): fp8_e4m3 is Hopper-only and fp8_e5m2 is rejected by vLLM's Gemma 4 attention assert. So A100 was running on roughly half the concurrent capacity of TPU's fp8 setup — and still beat TPU on medium and long throughput. On Hopper (H100) with fp8_e4m3, the gap could grow further. Conversely, TPU v6e — running aggressive fp8 KV compression — still beats A100 on short and on TPOT across the board.

Workload-based recommendation

Mapping the numbers to product use cases:

Use case	Profile	Recommendation	Rationale
Chat (interactive UX)	short (~1K in / ~256 out)	TPU v6e-4	Higher saturation throughput and lower TPOT → fast, smooth replies
RAG / mid-length generation	medium (~4K in / ~512 out)	Either	Saturation throughput is close. TPU if TPOT matters; A100 if cost is closer
Long-form summarization / code	long (~8K in / ~1K out)	A100 80GB × 2	1.44× higher throughput on long prefill — though TPU still has smoother token streaming
Streaming UX above all	any profile	TPU v6e-4	TPOT 2.5–5.8× faster than A100

Operational notes: where TPU and GPU each tripped us up

We hit multiple snags on the way to a clean run. Interestingly, the layers that broke on TPU and on GPU were quite different.

TPU side

Don't install vLLM via pip on TPU. PyPI's plain vllm is the CUDA build; the server fails at startup with Failed to infer device type. The official Docker image (vllm/vllm-tpu:gemma4) is far simpler.
Image entrypoint quirks. vllm/vllm-tpu:gemma4 uses a pass-through entrypoint, so the first arg has to be a runnable command. We put vllm serve <model> first; passing --model X directly produces bash: exec --: invalid option.
Pick the right TPU runtime. v2-alpha-tpuv6e. Confirm with gcloud compute tpus tpu-vm versions list --zone=$ZONE. TPU 7x (Ironwood) wasn't accessible from our project — application is in flight.
TPU v6e capacity is tight. Even with unlimited quota, Insufficient capacity was the most common failure; us-central1-{a,c} and us-east5-a were all capacity-constrained, and we eventually landed on us-east5-b. Queued Resource (QR) is a practical fallback if every zone fails on the first try.

GPU side

The GPU side hit fewer accelerator-specific issues; most of the friction was Linux distro and GCP resource management.

Fresh GCP projects start with zero GPU quota. The project we used had NVIDIA_A100_80GB_GPUS and NVIDIA_H100_80GB_GPUS at 0 in every region. We submitted three quota requests — the first two went out for Committed NVIDIA H100 GPUs (CUD), which were approved but useless for on-demand. We then requested both plain NVIDIA X 80GB GPUs (without the Committed or Preemptible prefix) and the new umbrella GPUS_PER_GPU_FAMILY (with gpu_family=NVIDIA_H100); we're still not sure exactly which combination is the bare minimum and would love expert input on this.
A100 had capacity issues too. Both us-east5-{a,b} returned ZONE_RESOURCE_POOL_EXHAUSTED. We followed GCP's hint and ended up in us-central1-c. (The TPU and GPU ended up in different regions; since the bench loops through docker exec on the same VM, we believe this doesn't affect comparison fairness.)
DLVM common-cu129-ubuntu-2404-nvidia-580 does not include Docker. It ships NVIDIA Driver 580 and NVIDIA Container Toolkit 1.17.8, but Docker engine itself you install yourself: sudo apt install -y docker.io followed by sudo nvidia-ctk runtime configure --runtime=docker.
A fresh VM has its apt held by unattended-upgrades. sudo apt install gives Could not get lock for the first few minutes. Our setup script now waits on the lock before proceeding.
Ubuntu 24.04 enforces PEP 668. pip install --user is blocked by externally-managed-environment. Tools like the HF CLI now go in a dedicated venv.
huggingface-cli is deprecated → hf CLI. As of huggingface_hub 1.12, huggingface-cli prints a "use hf instead" warning and stops working. The replacements are hf auth login --token and hf download REPO_ID.
HF cache permission collision. Once Docker (running as root) writes into ~/.cache/huggingface, the host user can't always write back: [Errno 13] Permission denied: ...lock. Fixed with sudo chown -R $USER:$USER ~/.cache/huggingface.
A100 + Gemma 4 can't use FP8 KV cache. Detailed above; the practical resolution is --kv-cache-dtype auto (= bf16).
Read vLLM v1 errors at the worker level. Engine core initialization failed. See root cause above. Failed core proc(s): {} is a messenger, not the actual exception. The real exception is in lines prefixed with (Worker_TP*). We extracted them with docker logs gemma4-gpu | grep '(Worker_TP'.

Cross-platform takeaways

Don't paste long shell commands through chat / IDE buffers. Backslash continuations and heredocs corrupt easily. Author them as files locally, scp them, then bash them.
Run long jobs under nohup or tmux/screen from the start. A single ssh disconnect cost us more than an hour of progress and accelerator time.

TPU breaks at the accelerator-specific layers (image, runtime, arg parsing, capacity). GPU breaks at the Linux distro and GCP-resource layers (quota, Docker, pip, hf). Plan for a half-day to one-day of buffer per platform on a fresh project.

Conclusion

Summary of findings

We benchmarked Gemma 4 31B on TPU v6e-4 and A100 80GB × 2 with vLLM, running 21 cases (3 profiles × 7 rates) on each side. Highlights:

Throughput winner depends on input/output length. TPU wins short. A100 wins medium and long.
TPOT is uniformly faster on TPU by 2.5–5.8×. For streaming UX, TPU has a real edge.
A100 ran with bf16 KV cache, not FP8. It still won on medium and long. On Hopper with fp8_e4m3, the gap could grow further.
The cleanest split: chat → TPU; long-form generation → A100.
Operationally, TPU and GPU break in different layers. Budget time accordingly on fresh projects.

For middle-of-the-road dense LLMs in the ~30B class on GCP, this hopefully gives a concrete reference for choosing TPU or GPU based on your actual workload shape.

Future work

H100 + FP8 KV cache for a fair fight. A100 was forced to bf16; H100 with fp8_e4m3 would let us put GPU on the same FP8 footing as TPU and rerun the sweep.
TPU 7x (Ironwood). 192 GiB HBM/chip — six times TPU v6e. Currently allowlist-only, but a follow-up benchmark is planned once it's broadly available.
Quantized weights. AWQ, GPTQ, and FP8 weight quantization for accuracy/throughput tradeoffs.
Larger and MoE models. Gemma 4 26B-A4B (MoE) and bigger dense models with quantization.

Appendix: Smoke test on the short profile

Before the main sweep, we ran a one-off smoke test on the short profile (input 1024 / output 256), submitting 1000 prompts with --request-rate inf-equivalent dispatch (Maximum request concurrency: None, Peak concurrent requests: 1000). The numbers aren't part of the official sweep but give a quick pre-flight comparison.

The authoritative numbers are in the Results section above. The smoke test is a single one-shot run right after Docker container startup, while the main sweep includes a warmup round before each of the 21 cases. The post-warmup A100 numbers come out slightly better (rate=∞: req/s 2.78 → 2.88, out_tok/s 712 → 737, median TTFT 174,897 → 162,543 ms; TPU is virtually unchanged). Treat the appendix as a "yes, both platforms are running the same configuration" sanity check rather than a benchmark snapshot.

Metric	TPU v6e-4 (FP8 KV)	A100 80GB × 2 (bf16 KV)	Ratio (TPU / A100)
Successful requests	1000	1000	—
Failed requests	0	0	—
Benchmark duration (s)	256	359	—
Request throughput (req/s)	3.91	2.78	1.40×
Output token throughput (tok/s)	1001	712	1.41×
Total token throughput (tok/s)	5004	3561	1.41×
Mean TTFT (ms)	121,590	176,526	TPU 1.45× faster
Median TTFT (ms)	120,927	174,897	TPU 1.45× faster
P99 TTFT (ms)	240,983	348,870	TPU 1.45× faster
Mean TPOT (ms)	46.8	112.3	TPU 2.40× faster
Median TPOT (ms)	44.8	114.7	TPU 2.56× faster
P99 TPOT (ms)	79.3	150.3	TPU 1.90× faster

Observations:

Throughput is ~1.4× higher on TPU. A100 is on bf16 KV cache (~half the concurrency of TPU's fp8), so this isn't a pure hardware gap.
TPOT is ~2× faster on TPU. KV cache size doesn't directly affect TPOT, so this is closer to a clean compute comparison and v6e looks strong. The main sweep confirms the pattern.
TTFT is in the 100-second range for both platforms. This is --request-rate inf with 1000 prompts thrown in at once — queue-dominated. In real Poisson traffic at lower rates, TTFT shrinks dramatically; the main sweep shows the realistic numbers at rate=1–32.

References

vLLM Documentation
vLLM Gemma 4 recipe
Gemma 4 31B Instruct on Hugging Face
Cloud TPU v6e (Trillium) docs
LLM Inference Hardware Benchmark (living hub) — this campaign indexed alongside all our other inference measurements

DEV Community: Sho Tanaka (tsho)

Turning Qwen2.5-0.5B into a JSON API with SFT — 23% 100% on a Free T4

I Tried Post-Training and All I Got Was "systemsystem"

Setup: Why a 0.5B Base Model

Seven Traps, in Order

Trap 1: KeyError: 'completion'

Trap 2: Output Collapses to "systemsystemsystem…"

Trap 3: RuntimeError — Tensors on cuda:1 and cuda:0

Trap 4: Full Fine-Tuning Collapses Again — Why I Switched to LoRA

Trap 5: Forgot model.eval() — a Fake Collapse

Trap 6: torch.cuda.is_bf16_supported() Returns True on a T4

Trap 7: Two Version-Drift Errors (BatchEncoding, torchao)

What the trap list actually shows

Experiment 1: What the Format Change Actually Looks Like

The part I got wrong at first: SFT made stopping worse

Experiment 2: Quantifying Success — a 0.5B JSON API

The Final Working Code

Reproduction Details

Discussion: SFT Isn't "Useless" — It Has a Different Job

Summary

References

pgvector vs LanceDB: A Reproducible Vector Search Benchmark (100k 1536-dim)

Introduction

The architectural difference, first

Environment & setup

Result 1: Ingest & index build — a LanceDB landslide

Result 2: The recall-latency curve — LanceDB 2× faster single-threaded

Result 3: Concurrency — pgvector flips it at 8 threads

Result 4: Filtered search — pgvector is "planner-dependent", LanceDB is steady

Which to pick

Summary

Future work

Benchmark code

References

Why the First Turn in a Coding Agent Can Use So Many Input Tokens — and Why That Gets Better Over Time

1. Why the First Turn Can Look Expensive

1-1. Why users notice high input token usage on the first turn

1-2. Why a short user prompt does not mean a small total prompt

2. What Gets Reused Across Turns

2-1. What makes up the reusable prefix: system prompt, tools, instructions, and history

2-2. The difference between cacheable context, cache writes, and cache hits

3. Why Cache Hit Rates Improve Over Time

3-1. Why cache hit rates usually improve after the first few turns

4. How to Improve Cache Reuse in Practice

4-1. How to improve cache hit rates without changing developer behavior too much

Switching models mid-session usually reduces cache reuse

4-2. Common misunderstandings about “high token usage on the first prompt”

5. How to Calculate and Monitor Cache Hit Rates

5-1. How cache hit rate is calculated

5-2. How to monitor cache hit rates in practice

6. Final Takeaway

References

TPU v6e vs A100 80GB 2 for Gemma 4 31B on vLLM: 21 Benchmarks Show When Each Wins

Introduction

Why "speed" and "cost" of LLM inference matter more than ever

Why vLLM

What this article covers

Environment & Setup

Hardware

Software

KV cache dtype: not symmetric between GPU and TPU

Model

Dataset

Methodology

How we drove the load

Sweep

Metrics

Results

Summary: saturation throughput

Output throughput vs. request rate

TPOT (per-output-token time) vs. request rate

TTFT (time to first token) vs. request rate (log scale)

Short profile (input 1024 / output 256)

Medium profile (input 4096 / output 512)

Long profile (input 8000 / output 1000)

Combined summary

Cost estimate ($ / 1M output tokens)

Discussion

The throughput winner switches with input/output length

TPOT is dramatically faster on TPU across every profile

Trap 1: `KeyError: 'completion'`

Trap 3: `RuntimeError` — Tensors on cuda:1 and cuda:0

Trap 5: Forgot `model.eval()` — a Fake Collapse

Trap 6: `torch.cuda.is_bf16_supported()` Returns True on a T4

Trap 7: Two Version-Drift Errors (`BatchEncoding`, `torchao`)