Every cluster procurement meeting in the last two years has had the same argument. InfiniBand is expensive, vendor-locked, and hard to hire for. Ethernet is cheap, ubiquitous, and every network engineer already knows it. So why do the largest training clusters still ship with InfiniBand, and why does the Ethernet camp keep promising that "this generation is different"?
The honest answer is that bandwidth is no longer the differentiator. Both fabrics deliver 400G and 800G per port. The differentiator is what the fabric does in the few hundred microseconds after a link saturates, and that behaviour is where the money actually goes.
Part One: What a training job actually asks of a network
A data-parallel training step ends with an all-reduce. Every GPU has a gradient tensor, and every GPU needs the sum of all of them. On 512 GPUs, a naive ring all-reduce moves roughly twice the tensor size per GPU. On a 70B model with 140 GB of gradients in FP16, that is not a message, it is a freight train.
The key property is that all-reduce is synchronised. Nobody can proceed until everybody finishes. This changes the economics of congestion completely. In a web service, a dropped or delayed packet costs one user some milliseconds and the system degrades gracefully. In an all-reduce, the slowest participant sets the pace for all 512 GPUs, and the cost of the straggler is multiplied by the whole cluster. A 3% tail-latency penalty on the fabric becomes a 3% penalty on a multi-million-dollar training run.
That is the asymmetry InfiniBand was built for. Its credit-based link-layer flow control means a sender never transmits unless the receiver has buffer space reserved. Congestion turns into backpressure, not drops. Adaptive routing spreads traffic across paths at the packet level. In practice this produces a latency distribution with a very short tail, which is exactly the metric that synchronised workloads care about.
Part Two: Why Ethernet's answer took so long
Standard Ethernet drops packets when a queue fills. For TCP that is fine, because TCP retransmits and the application never notices. For RDMA over Converged Ethernet, a drop is catastrophic: the receiving NIC has already consumed the packet sequence and cannot reorder, so a single lost packet can stall a queue pair until a timeout fires.
RoCEv2 solved the drop problem with Priority Flow Control, which pauses a traffic class when buffers fill. It works, and it introduced a new failure mode that operators still fight: pause frames propagate. One congested port can pause its neighbour, which pauses its neighbour, and a single hot spot can freeze a large section of the fabric. Then came Explicit Congestion Notification, then dynamic routing, then the Ultra Ethernet Consortium's whole reason for existing.
The point is not that Ethernet cannot do this. The point is that Ethernet's congestion control was designed to be forgiving, and training workloads need it to be precise — every fix since 2019 has been an attempt to retrofit precision onto a forgiving design.
Part Three: The cost accounting nobody puts on the slide
A 512-GPU cluster with 400G InfiniBand costs roughly $1,200 to $1,800 per GPU in switching, cabling and optics at current list prices. The Ethernet equivalent with comparable non-blocking topology lands nearer $700 to $1,000. On a $15 million cluster, that is a $250,000 to $400,000 difference, which sounds decisive until you price the alternative.
If the Ethernet fabric loses 4% of training throughput to tail latency and retransmission overhead, the same cluster finishes 4% fewer runs per year. At a fully loaded GPU-hour cost of $2.50 and 8,000 GPU-hours per month per GPU, 4% is about $48,000 per month, or $576,000 a year. The cheaper fabric is only cheaper if you can hold the tail latency within roughly two percentage points of InfiniBand, and that is a software engineering problem you have to staff.
This is why the market split looks the way it does. Inference clusters, which are request-parallel and latency-tolerant, moved to Ethernet quickly. Training clusters, which are synchronised and latency-intolerant, stayed. The Ultra Ethernet Consortium's 1.0 specification is the industry's bet that this split is temporary, and the first real deployments are only now appearing.
Part Four: What changes by 2029
Three things are happening at once, and they point in the same direction.
First, optics are moving into the package. Co-packaged optics removes the pluggable module from the power and signal-integrity budget, which lowers the per-port cost of very high radix switching. When a 512-port switch costs 30% less, the fabric premium shrinks in absolute terms.
Second, the workload is changing. Mixture-of-experts training and disaggregated inference both reduce the frequency of global synchronisation. Fewer synchronised barriers means the tail latency penalty shrinks, which weakens InfiniBand's core advantage.
Third, and least discussed, the operational cost is shifting. A team that runs Ethernet can hire from a pool of hundreds of thousands of engineers. A team that runs InfiniBand hires from a pool of thousands. At scale, the salary and on-call burden of a specialised fabric team can exceed the hardware premium.
My estimate: by 2029 the AI fabric market splits roughly 55% Ethernet, 35% Ultra Ethernet, 10% InfiniBand, with InfiniBand retaining the very largest and most latency-sensitive training clusters. The total fabric market grows from about $8 billion today to $19 to $24 billion, because port count grows faster than price per port falls.
The daily-life angle
This sounds like a data-centre problem, but it reaches ordinary users through the price and availability of model features. Every percentage point of fabric efficiency is a percentage point of cost per token. When a chat assistant answers in 400 milliseconds instead of 600, or when a video model renders a clip overnight instead of over a weekend, that is the fabric budget being spent well. Conversely, the reason some features stay behind a premium tier is often not the model, it is the interconnect that the model needs to run at scale.
For engineers, the practical lesson is narrower and more useful: measure your tail, not your average. A fabric that delivers 95% of line rate on average and 60% at p99 will destroy a synchronised workload while looking perfect on a dashboard. If you cannot measure p99 across the whole fabric, you are not buying a network, you are buying a number on a slide.
The transferable lesson
Every infrastructure decision eventually reduces to the same question: which failure mode can your workload tolerate? Ethernet tolerates loss and pays in tail latency. InfiniBand refuses loss and pays in cost and lock-in. Neither is wrong; they are optimised for different shapes of work.
The mistake is choosing on price per port and discovering the workload's real constraint eighteen months later, when the cluster is full and the retraining budget is gone. Buy the failure mode that matches your job, then optimise the price.
SOL: HCqPmPJM3NVtS6YrmVaTZYnuP8XZ2PFgQZKT94VK1LkS
Creating this takes real effort — thank you for the support.
Top comments (0)