DEV Community

Douglas Rawson
Douglas Rawson

Posted on

Distributed Training Across Mixed GPUs: Solving the Heterogeneous Fleet Problem

Distributed Training Across Mixed GPUs: Solving the Heterogeneous Fleet Problem

As machine learning models grow larger, the hardware requirements become more demanding. But what if your lab has a mix of GPUs from different generations — an RTX 3090 here, a V100 there, maybe even some older M40s gathering dust? Traditionally, distributed training tools assume homogeneous hardware, leaving these mismatched cards underutilized.

The Challenge

Most distributed training frameworks expect identical GPUs across nodes. If your setup includes:

  • NVIDIA RTX 3090 (24GB VRAM)
  • RTX 4090 (24GB VRAM)
  • Tesla V100 (16GB VRAM)
  • Quadro M40 (24GB VRAM)

You can't easily pool them into a single training job. The differences in architecture, memory, and compute capability create bottlenecks.

A New Approach

We're experimenting with a distributed training method that works across heterogeneous GPU fleets. The key components:

4-Bit NF4 Quantized Sharding

  • Uses 4-bit quantization with Normal Float 4 (NF4) distribution for efficient memory usage
  • Shards model weights across GPUs regardless of their specs
  • Balances load dynamically based on each GPU's capabilities

WireGuard Mesh Networking

  • Creates a secure, peer-to-peer mesh between machines
  • Works over regular Ethernet (1GbE or faster)
  • Minimal latency overhead for inter-GPU communication

Why This Matters

This approach enables:

  • Utilizing legacy hardware alongside modern GPUs
  • Scaling training without buying matching equipment
  • Cost-effective expansion of ML infrastructure
  • Research flexibility for teams with varied hardware

We're Looking for Feedback

We're running a free 4-week beta to validate this approach. If you have a messy GPU setup and want to test distributed training across them, we'd love your input.

Beta Signup: https://shardpool.aurora-sentient.net/

Share your thoughts in the comments — what's your biggest hardware heterogeneity challenge?

Top comments (0)