DeepSpeed

Distributed training and inference library. Introduced ZeRO (Zero Redundancy Optimizer) for memory-efficient data parallelism, 3D parallelism (data + tensor + pipeline), and inference optimizations. 42K+ GitHub stars.

DeepSpeed was used to train BLOOM-176B, Megatron-Turing NLG 530B, and many other frontier models. Its ZeRO optimizer enabled training models that otherwise wouldn't fit in GPU memory. MIT License.