Most inference benchmarks look like a chatbot. One question, one answer, next.
Agents don't work like that.
A coding agent runs a long session. It keeps a huge shared context. It spins up subagents. It pauses, thinks, comes back, and does it again. In our run, the average request carried 87K to 106K input tokens.
That's what SemiAnalysis AgentX simulates. It's the closest public benchmark to what our customers' traffic looks like. So it's the one I care about most.
The setup
- Model: Qwen3.8-Flash-Next
- Hardware: 8× NVIDIA H100
- Backboard: BF16, full precision
- SGLang: the published AgentX reference, at FP8
- Load: 1, 4, 8, 12 and 16 live agent sessions
Read that precision line twice. SGLang ran at FP8. We ran at BF16. They took the shortcut. We didn't. And we still won.
Interactivity
Interactivity is how fast tokens stream to each user once the answer starts. For an agent, that's how fast it gets through its work.
| Sessions | p90 interactivity vs SGLang |
|---|---|
| 1 | 12.8× |
| 12 | 7.0× |
| 16 | 5.0× |
Time to first token
| Sessions | p90 TTFT, faster by |
|---|---|
| 1 | 13.6× |
| 12 | 3.1× |
| 16 | 2.5× |
Ahead at every session count we tested.
Throughput
Responsiveness didn't cost us throughput. Output throughput per GPU came in up to 2.2× SGLang's. Total throughput per GPU up to 2.1×.
Faster for each agent. More total work from the same eight cards.
What our engineers found along the way
One of our engineers spent a long stretch trying to work out why SGLang's decode speed was beating ours in some of his tests. She went through our engine line by line.
It wasn't us. SGLang was applying simulated acceptance in speculative decoding the wrong way. In her testing it pushed their tokens per second up by around 8%.
Then yesterday AgentX republished its SGLang numbers with the fix in. They came down about 3%.
Our comparison above was against the earlier reference. The correction moves their side of the ratio down. Not ours.
Read the full report
Everything is in the AgentX report on our blog.
Running agents on your own GPUs? Let's see what this does for yours.
Top comments (0)