I test software for a living. So when a vendor calls an AI model "fast," I don't trust the word. I measure it.
Most leaderboards rank how smart a model is. Almost none rank how fast it answers. You pick a model because it scored well, ship it, and then your users sit and wait.
Speed is two different numbers. People mix them up constantly.
The two numbers
Time to first token (TTFT). The wait before the first word appears. You feel this every time a chatbot "thinks" before replying.
Tokens per second (TPS). How fast the model writes once it starts. A token is a chunk of a word.
A model can be great at one and terrible at the other. You need both.
How I measure it
I run an independent tracker called ollamatps.com. It benchmarks 42 Ollama Cloud models. Here is the exact method, because a benchmark you cannot inspect is just a claim.
- One fixed prompt. Every run asks the model to write a 400-word explanation of HTTP routing. Same prompt, every model, every time.
-
Fixed output cap.
max_tokensis capped at 300. It never changes between runs. - Every ~10 minutes. Each model is re-tested continuously, not once.
- TTFT is measured from the moment the request is sent to the first non-empty content chunk. It includes network round-trip and prompt-processing time. That is honest, because it is what you actually wait.
-
TPS is generation throughput only:
eval_count / (total_duration - time_to_first_token). The startup wait is removed, so TPS measures pure writing speed.
Same prompt, same cap, same schedule. That is what makes two models comparable.
Building this was a testing job, not a coding job. Retries on failure. A reliability score per model. A circuit breaker for models that keep failing. If you cannot trust the measurement, the number is noise. That part is the same work I do on any test system.
What surprised me
Bigger is not faster.
The fastest model on the board is one of the smallest: a 30B model at over 200 tokens per second. The model literally named "ultra" is dead last, under 8 tokens per second.
And the wait varies wildly. TTFT ranges from about 0.3 seconds to 23 seconds across the 42 models. Same cloud. Roughly 80x difference in how long you wait for the first word.
If you picked your model on a benchmark score alone, you have no idea which of these you are getting.
Why I rebuilt it
The first version tracked fewer models and was less robust. I rebuilt the engine this month (v2) to be multi-provider and to test continuously. The live board updates every 10 minutes.
Watch it run: ollamatps.com
I'm Anton. I build AI tools and measure how fast they really run. Former Apple engineer. I make ollamatps.com and write about building and measuring AI at anton.qa.
Top comments (1)
Your article is interest for me.
Are you AI - developer?
I've new idea.
if you open to it, we can meet on telegram.
telegram: @venus0427
thanks for nice article.