AICPLIGHT

Posted on Aug 28

InfiniBand for AI Clusters: Architecture, RDMA, and Optical Connectivity Explained

#ai #infiniband #networking #gpu

As AI clusters continue to scale, networking is becoming one of the most important factors determining overall system performance.

A GPU cluster may contain hundreds or even thousands of GPUs, but these GPUs cannot work efficiently in isolation. During distributed AI training, they constantly exchange gradients, parameters, and synchronization data.

This creates a simple but important question for infrastructure engineers:

How do you build a network that can keep thousands of GPUs communicating without becoming the bottleneck?

InfiniBand is one of the most widely deployed answers.

Originally designed for high-performance computing (HPC), InfiniBand has become an important networking technology for large-scale AI infrastructure because it combines high bandwidth, low latency, RDMA support, and centralized fabric management.

This article looks at how an InfiniBand network works, what hardware is required, and what engineers should consider when designing an AI cluster.

1. Why AI Clusters Need High-Performance Networking

Traditional enterprise applications usually generate relatively independent network traffic.

AI training is different.

In distributed training, multiple GPUs participate in the same computation. They need to exchange information continuously during operations such as gradient synchronization and collective communication.

For example, an AI training job may perform an AllReduce operation in which GPUs exchange and aggregate data across the cluster.

The larger the cluster becomes, the more important network performance becomes.

A slow or congested network can cause GPUs to wait for communication instead of performing computation.

That means adding more GPUs does not necessarily produce proportional performance improvements.

The network must scale together with compute.

This is one of the main reasons high-performance networking technologies such as InfiniBand are widely used in AI and HPC environments.

2. What Makes InfiniBand Different?

InfiniBand is a high-speed networking architecture designed for low-latency and high-throughput communication.

One of its biggest advantages is native support for Remote Direct Memory Access (RDMA).

With conventional networking, data typically passes through the operating system and CPU processing stack.

RDMA changes this model by allowing data to move directly between the memory of two devices.

This reduces:

CPU involvement
Memory-copy operations
Software overhead
Communication latency

For AI workloads, this is particularly important because network communication happens continuously during distributed training.

Instead of using CPU resources to manage every communication operation, the network adapter can handle much of the data movement directly.

3. RDMA: Why It Matters for GPU Communication

The key idea behind RDMA is simple:

Move data directly between device memories while minimizing CPU and operating-system involvement.

A simplified communication path looks like this:

Traditional Networking

Application
     ↓
Operating System
     ↓
CPU Processing
     ↓
Network Stack
     ↓
Network Adapter
     ↓
Network

With RDMA, the path can be significantly more efficient:

RDMA

GPU / Device Memory
        ↓
   RDMA NIC
        ↓
     Network
        ↓
   RDMA NIC
        ↓
GPU / Device Memory

This architecture is particularly useful for tightly coupled workloads.

In AI training, thousands of GPUs may exchange data simultaneously. Reducing communication overhead helps improve GPU utilization and makes cluster performance more predictable.

4. The Main Components of an InfiniBand Fabric

An InfiniBand network is not simply a collection of switches and cables.

A complete fabric typically includes:

InfiniBand adapters
InfiniBand switches
Subnet Manager
InfiniBand cables
Optical transceivers

Each component has a specific role.

InfiniBand Network Adapters

The network adapter connects a GPU server to the InfiniBand fabric.

NVIDIA ConnectX adapters are widely used in modern AI infrastructure.

For example, ConnectX-7 supports 400G connectivity and can operate with both InfiniBand and Ethernet, providing flexibility for different deployment scenarios.

The adapter also handles RDMA and hardware-level traffic processing, reducing the workload placed on the host CPU.

One important deployment consideration is PCIe compatibility.

For example, pairing a high-speed adapter with an insufficient PCIe interface can prevent the adapter from reaching its full potential.

Therefore, the server motherboard, PCIe generation, GPU architecture, and NIC should always be evaluated as a complete system.

5. InfiniBand Switches and Fabric Management

The switch is responsible for forwarding traffic between nodes.

Modern AI clusters commonly use NVIDIA Quantum platforms for InfiniBand networking.

Unlike conventional Ethernet networks that rely heavily on distributed routing protocols, InfiniBand uses a Subnet Manager (SM) to manage the fabric.

The Subnet Manager performs tasks such as:

Device discovery
Route calculation
Forwarding-table configuration
Quality-of-Service configuration
Partition management
Network recovery

This centralized approach helps maintain predictable communication paths across the fabric.

For large AI clusters, predictable behavior is particularly important because communication patterns can generate substantial amounts of east-west traffic.

6. InfiniBand Network Topology

A common architecture for large AI clusters is the Spine-Leaf topology.

A simplified design looks like this:

                 Spine Layer
          ┌────────┼────────┐
          │        │        │
       Spine 1  Spine 2  Spine 3
          │        │        │
       ───┼────────┼────────┼───
          │        │        │
        Leaf 1   Leaf 2   Leaf 3
       /  |  \   / | \   / |  \
     GPU GPU GPU GPU GPU GPU GPU

The leaf switches connect GPU servers, while spine switches provide connectivity between leaf switches.

This architecture offers several advantages:

Scalable bandwidth
Predictable paths
High port utilization
Simplified expansion
Efficient east-west communication

As the number of GPU nodes increases, additional leaf and spine switches can be added to expand the fabric.

7. InfiniBand Speed Evolution

One of the most important trends in InfiniBand is the rapid increase in port bandwidth.

The technology has evolved through multiple generations:

Generation	Approx. Port Rate
SDR	10G
DDR	20G
QDR	40G
FDR	56G
EDR	100G
HDR	200G
NDR	400G
XDR	800G
GDR	1.6T

The increase is not simply about faster signaling.

Higher-speed InfiniBand also requires corresponding changes in:

Switch ASICs
Network adapters
Optical transceivers
Cables
SerDes technology
Thermal design

For AI infrastructure engineers, this means a network upgrade should be considered as an end-to-end architecture rather than a simple switch replacement.

8. Optical Connectivity Is a Critical Part of the Fabric

It is easy to focus on GPUs and switches when designing an AI cluster.

But the physical interconnect layer is equally important.

High-speed optical modules provide the links between network adapters and switches, and between switches themselves.

For modern InfiniBand deployments, engineers may encounter 400G NDR and 800G XDR optical connectivity.

The optical module must match the required:

InfiniBand generation
Port speed
Form factor
Fiber type
Transmission distance
Switch/NIC compatibility

For example, an 800G InfiniBand deployment requires optical components designed for the corresponding InfiniBand application. An Ethernet optical module should not automatically be assumed to work simply because it has the same nominal data rate.

This distinction becomes increasingly important as Ethernet and InfiniBand both move toward 800G and 1.6T connectivity.

9. Multimode vs. Single-Mode Fiber

The choice of optical technology also depends heavily on transmission distance.

Multimode Fiber

Multimode solutions are generally suitable for shorter-distance connections.

Typical applications include:

Server-to-leaf connections
Short intra-row links
Short inter-rack connections

They can provide a cost-effective solution where distances are limited.

Single-Mode Fiber

Single-mode solutions are more appropriate for longer-distance connections.

They can be used for:

Longer switch-to-switch links
Inter-room connections
Large-scale data center fabrics

The selection should consider both distance and total deployment cost.

Power consumption is another important factor.

Thousands of optical modules can be installed in a single AI cluster, so even a small difference in power consumption per module can become significant at the rack or data-center level.

10. DAC vs. AOC vs. Optical Transceivers

Not every InfiniBand connection requires a pluggable optical module.

The appropriate interconnect depends on distance and deployment requirements.

DAC

Direct Attach Copper is generally suitable for short connections.

Typical applications include:

Connections inside the same rack
GPU server to leaf switch
Short-distance switch connections

DAC can provide a lower-cost solution for short links.

AOC

Active Optical Cables integrate optical components into the cable assembly.

They are useful when:

The distance is longer than practical DAC deployments
Lower cable weight is desirable
Flexible optical connectivity is needed

Pluggable Optical Modules

For longer-distance connections and scalable switch fabrics, pluggable optical transceivers provide greater flexibility.

They allow the network designer to select different fiber types and transmission distances without replacing the entire cable assembly.

The key principle is:

Choose the interconnect based on distance, bandwidth, density, power, and deployment environment—not simply the nominal data rate.

11. How Many Optical Modules Does an AI Cluster Need?

This is where AI networking becomes especially interesting for infrastructure planning.

Consider a large GPU cluster based on a Spine-Leaf InfiniBand architecture.

A reference deployment described by NVIDIA includes:

127 H100 servers
1,016 GPUs
32 Leaf switches
16 Spine switches

The optical connectivity requirement can quickly reach thousands of modules. AICPLIGHT's analysis estimates approximately 2,421 800G optical modules for this configuration under the stated assumptions.

The calculation illustrates why optical connectivity should be planned together with GPU capacity.

For example, the GPU-to-optical-module ratio can be around 1:2.38 in this architecture.

That means a cluster with thousands of GPUs may require several thousand high-speed optical components.

As clusters scale toward 10,000 GPUs and beyond, the optical layer becomes a major part of both the infrastructure design and the deployment budget.

12. Common InfiniBand Deployment Mistakes

Building a high-speed InfiniBand network is not simply a matter of purchasing the fastest available hardware.

Several practical issues need to be considered.

Mistake 1: Mixing Incompatible Optical Modules

A module with the correct data rate is not necessarily compatible with the intended InfiniBand application.

Always verify:

Protocol compatibility
Form factor
Port type
Optical specification
Switch/NIC support

Mistake 2: Ignoring PCIe Limitations

A high-speed NIC cannot deliver its full performance if the server's PCIe interface becomes the bottleneck.

NIC, server motherboard, PCIe generation, and GPU platform must be evaluated together.

Mistake 3: Choosing the Wrong Fiber Type

Short-distance and long-distance links have different requirements.

Using single-mode optics where multimode connectivity is sufficient may increase cost unnecessarily, while using short-reach multimode solutions for long-distance links can create deployment problems.

Mistake 4: Underestimating Optical Module Quantities

Network planning should not stop at the switch count.

Engineers should calculate:

GPU-to-leaf connections
Leaf-to-spine connections
Redundant links
Spare modules
Cable requirements

A small error in the initial calculation can become a significant procurement issue when multiplied across thousands of links.

13. InfiniBand vs. Ethernet: Is InfiniBand Always Better?

Not necessarily.

Ethernet has a much broader ecosystem and supports a huge range of enterprise workloads.

Technologies such as RoCE allow Ethernet networks to support RDMA-based communication and are increasingly being considered for AI infrastructure.

The choice depends on the workload and deployment requirements.

InfiniBand is particularly attractive when the primary objective is:

Maximum AI training performance
Predictable low latency
Large-scale GPU communication
Dedicated HPC infrastructure
High-performance collective operations

Ethernet-based architectures may be attractive when organizations prioritize:

Existing Ethernet infrastructure
Broader interoperability
Network convergence
Operational familiarity

The right decision should therefore be based on workload characteristics, scale, operational requirements, and long-term architecture.

14. What Comes After 800G?

AI networking is moving rapidly toward even higher bandwidth.

NDR 400G has already become an important generation for AI clusters, while XDR 800G is designed for the next level of scale.

Beyond 800G, 1.6T-class connectivity will become increasingly important as GPU performance continues to increase.

This evolution will affect the entire network stack:

Faster GPUs
     ↓
More GPU-to-GPU Traffic
     ↓
Higher Network Bandwidth
     ↓
800G / 1.6T Interconnects
     ↓
Higher-Speed Optics
     ↓
New Thermal & Power Challenges

The important point is that networking cannot evolve independently from computing.

As GPU performance increases, the network must evolve at a similar pace to prevent communication from becoming the limiting factor.

Conclusion

InfiniBand has become an important technology for large-scale AI infrastructure because it was designed around the requirements of high-performance computing.

Its combination of RDMA, low latency, high bandwidth, centralized fabric management, and scalable switching makes it particularly suitable for tightly coupled GPU workloads.

However, building an effective InfiniBand fabric requires more than selecting a high-speed switch.

Engineers need to consider the complete infrastructure:

GPU → NIC → Leaf → Spine → Optical Module → Fiber/Cable

Every component must be matched in terms of speed, protocol, form factor, distance, power, and compatibility.

As AI clusters move from hundreds to thousands and eventually tens of thousands of GPUs, these considerations will become increasingly important.

The future of AI networking will likely involve 800G, 1.6T, and even higher-speed interconnects, but the fundamental principle will remain the same:

The network must scale with the compute.

DEV Community