Pytorch Distributed Overview

Overview for the torch.distributed package.

refs

• DDP (Distributed Data Parallel)
• FSDP (Fully Shared Data Parallel Training)
• TP (Tensor Parallel)
• PP (Pipeline Parallel)