Tokens per second told me nothing about agent performance

I benchmarked a dozen free AI APIs on generation throughput, then plugged the same models into an agent harness and gave them real tasks. The rankings barely overlapped.

Two examples from the same platform:

• Gemma 4 31B — 50.9 tok/s, second fastest I measured. Inside the agent: hung until the 900-second timeout. Never finished.
• DeepSeek V4 Flash — 27.3 tok/s, the slowest of the group. Inside the agent: completed the task correctly.

And Nemotron Super 120B, which swung between 27 and 100 tok/s across runs of the streaming benchmark, was the fastest model in the agent loop at 43 seconds.

It makes sense once you look at what each number measures. Tokens per second is typing speed. Agent wall-clock time is decision quality — how many tool calls it makes, how many wrong turns it takes, whether it re-reads a file it already understood. A verbose fast model loses to a terse slow one.

So if you're picking a model for agent work, throughput leaderboards are close to useless. Twenty minutes with your actual task will tell you more.

Method and full numbers: https://toolfreebie.com/free-ai-models-agent-test/