subcluster
Distributed Runs
3 articles
Articles
May 22, 2026
Multi-GPU Tensor Parallelism Setup: Configuration and Optimization Guide
A 70B model needs about 140GB in FP16 and does not fit on one 80GB GPU. Tensor parallelism splits weight matrices across devices, at the cost of four all-reduce collectives per transformer layer in a training step.
May 16, 2026
Multi GPU Distributed Training Setup Guide: Frameworks & Infrastructure
Scaling from a single GPU to a multi-node cluster introduces complex communication bottlenecks and fatal memory errors. Learn how to configure DDP, FSDP, and DeepSpeed while optimizing your infrastructure for maximum throughput.
February 23, 2026
ZeRO-3 vs FSDP: A Deep Dive into Memory Efficiency for LLMs
Scaling large language models requires moving beyond standard data parallelism to overcome the memory wall. This technical guide compares DeepSpeed ZeRO-3 and PyTorch FSDP to help engineers optimize GPU utilization and eliminate out-of-memory errors.