Distributed and Multi-GPU Training

PyTorch Distributed and Multi-GPU Training

• Multi Node PyTorch Distributed Training Guide For People In A Hurry - Chuan Li - August 26, 2022 - amazing, and very condensed
• PyTorch Distributed Training - Lei Mao
• PyTorch Distributed Evaluation - Leo Mao
• WRITING DISTRIBUTED APPLICATIONS WITH PYTORCH - intro, PyTorch official, worth reading (written by Séb Arnold)
• pytorch/torch/distributed/ as of torch v2.0.1 bugfix

Backends

• NCCL documentation
• Gloo and gloo/docs