| | --- |
| | title: Zero Bubble Pipeline Parallellism |
| | emoji: 🏆 |
| | colorFrom: indigo |
| | colorTo: red |
| | sdk: gradio |
| | sdk_version: 4.36.1 |
| | app_file: app.py |
| | pinned: false |
| | license: apache-2.0 |
| | --- |
| | |
| |
|
| | # Zero Bubble Pipeline Parallelism |
| |
|
| | Zero Bubble Pipeline Parallelism is a novel pipeline parallelism algorithm able to reduce the bubble of pipeline parallelism to almost zero while preserving synchronous semantics. |
| |
|
| | Check out our paper at: |
| | * [Arxiv Version with ZBV](https://arxiv.org/abs/2401.10241) |
| | * [ICLR Accepted version with ZB1P and ZB2P](https://openreview.net/pdf?id=tuzTN0eIO5) |
| |
|
| | Try out our implementation based on Megatron on [https://github.com/sail-sg/zero-bubble-pipeline-parallelism](https://github.com/sail-sg/zero-bubble-pipeline-parallelism) |
| |
|
| | Experiments shows zero bubble pipeline parallelism can accelerate training up to 30% with a similar memory comsumption. A detailed table of experiments is coming soon. |
