README.md

Distributed training with Microsoft DeepSpeed

Author: Zhen Xie zhen.xie@anl.gov

DeepSpeed is a deep learning optimization library that makes distributed training easy, efficient, and effective.

Here is the DeepSpeed implementation: https://github.com/microsoft/DeepSpeed

And their paper about this:

1. Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, Yuxiong He. (2019) ZeRO: memory optimizations toward training trillion parameter models. arXiv:1910.02054 and In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC '20).
2. Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. (2020) DeepSpeed: System Optimizations Enable Training Deep Learning Models with Over 100 Billion Parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD '20, Tutorial).
3. Minjia Zhang, Yuxiong He. (2020) Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping. arXiv:2010.13369 and NeurIPS 2020.

How does DeepSpeed work?

DeepSpeed delivers extreme-scale model training for everyone, from data scientists training on massive supercomputers to those training on low-end clusters or even on a single GPU:

• Extreme scale: Using current generation of GPU clusters with hundreds of devices, 3D parallelism of DeepSpeed can efficiently train deep learning models with trillions of parameters.
• Extremely memory efficient: With just a single GPU, ZeRO-Offload of DeepSpeed can train models with over 10B parameters, 10x bigger than the state of arts, democratizing multi-billion-parameter model training such that many deep learning scientists can explore bigger and better models.
• Extremely long sequence length: Sparse attention of DeepSpeed powers an order-of-magnitude longer input sequence and obtains up to 6x faster execution comparing with dense transformers.
• Extremely communication efficient: 3D parallelism improves communication efficiency allows users to train multi-billion-parameter models 2–7x faster on clusters with limited network bandwidth. 1-bit Adam/1-bit LAMB reduce communication volume by up to 5x while achieving similar convergence efficiency to Adam/LAMB, allowing for scaling to different types of GPU clusters and networks.

Early adopters of DeepSpeed have already produced a language model (LM) with over 17B parameters called Turing-NLG, establishing a new SOTA in the LM category.

DeepSpeed is an important part of Microsoft’s new AI at Scale initiative to enable next-generation AI capabilities at scale, where you can find more information here.

Setting up DeepSpeed

The quickest way to get started with DeepSpeed is via pip, this will install the latest release of DeepSpeed which is not tied to specific PyTorch or CUDA versions. DeepSpeed includes several C++/CUDA extensions that we commonly refer to as our 'ops'. By default, all of these extensions/ops will be built just-in-time (JIT) using torch's JIT C++ extension loader that relies on ninja to build and dynamically link them at runtime.

Note: PyTorch must be installed before installing DeepSpeed.

pip install deepspeed

After installation, you can validate your install and see which extensions/ops your machine is compatible with via the DeepSpeed environment report.

ds_report

If you would like to pre-install any of the DeepSpeed extensions/ops (instead of JIT compiling) or install pre-compiled ops via PyPI please see our advanced installation instructions

Example

There is an example deployed by DeepSpeed here, with PyTorch using CIFAR-10.

For interactive job:

It has two steps:

Step 1: "Setting up env"

conda env create --name deepspeed --file /lus/theta-fs0/projects/datascience/zhen/env_deepspeed.yml //set up env and install packages

conda activate deepspeed // activate env

Step 2: "Run script"

cd /lus/theta-fs0/projects/datascience/zhen/DeepSpeed

deepspeed cifar10_deepspeed.py --deepspeed --deepspeed_config ds_config.json $@

For submitting jobs in the script (non-interactive) job mode, take a look in the submissions/ folder for more details about this.

DeepSpeed Performance

• DeepSpeed powers 8x larger MoE model training with high performance
  • Mixture of Experts (MoE) tutorial.
• Curriculum learning: a regularization method for stable and 2.6x faster GPT-2 pre-training with 8x/4x larger batch size/learning rate
• DeepSpeed: Accelerating large-scale model inference and training via system optimizations and compression
• 1-bit LAMB: up to 4.6x less communication and 2.8x faster training, together with LAMB's convergence speed at large batch sizes
• ZeRO-Infinity unlocks unprecedented model scale for deep learning training
  • Tutorial on how to use different stages of ZeRO
• [DeepSpeed on AzureML] Transformers and CIFAR examples are now available on AzureML GitHub
• [PyTorch Lightning Blog] Accessible Multi-Billion Parameter Model Training with PyTorch Lightning + DeepSpeed
• 1-bit Adam v2: NCCL-based implementation and more
• ZeRO-3 Offload: Scale your models to trillion parameters without code changes while leveraging both CPUs & GPUs
• [Hugging Face Blog] Fit More and Train Faster With ZeRO via DeepSpeed and FairScale