Memory Capacity Binds Before FLOPs Do
In agentic inference, memory capacity binds before FLOPs do. That is the first finding from AgentX, the open-source benchmark SemiAnalysis built for replaying real agentic coding traffic at one million context.
The numbers on DeepSeek V4 make the point. The HBM working set decided the KV-cache hit rate: 43 million tokens and 91 percent on a B300, against 22 million and 73 percent on a B200. The B300 run used 384 concurrent traces, the B200 run 196. Same model, same task, different memory budget, and the hit rate moved by eighteen points.
Why the hit rate matters: agentic sessions reuse prefixes. Every turn builds on the context before it, so most of the context can be served from the KV cache rather than recomputed. A miss means re-prefilling context you have already paid for once. In an agent loop of dozens of sequential calls, those misses compound into wall-clock time and compute you cannot get back.
AgentX matters because it measures the right thing. It replays 393 anonymised internal Claude Code traces, structure and timing preserved, under Apache 2.0. Fixed-sequence benchmarks reflect chip and kernel performance; agentic workloads reflect the systems problem: KV tensors, routing, and offload across memory tiers. The first open-source benchmark of this shape has produced the finding the labs have been converging on: the binding constraint in agentic inference is not peak FLOPs, it is the size of the memory working set.
The practical rule for anyone sizing an inference stack: size working set before FLOPs. A spec sheet that leads with peak throughput hides the number that actually determines your agent's latency. Measure step latency at your own concurrency, including prefill, before you trust a batch-of-one headline.
The pattern is durable. When compute gets cheap enough, the constraint migrates to the layer beneath it, the same migration the industry has watched in every previous scaling phase. AgentX is the first open benchmark to show where it has landed in agentic inference: not in the silicon that generates tokens, but in the memory that holds the conversation.
Top comments (0)