AI Architecture & Systems
1
Microarchitecture
2
Kernel
3
Compiler
4
Architecture
5
System
6
Algorithms
System
/
§5.8
Pipeline Parallelism
5.8.1
Stage Partitioning and Microbatches
#
5.8.2
GPipe, 1F1B, and Interleaved Pipelines
#
5.8.3
Pipeline Bubbles and Load Balance
#
5.8.4
Zero-Bubble Scheduling and Execution Dependencies
#
5.8.5
Activation Communication and Cross-Stage Overlap
#
5.8.6
Virtual Pipeline Stages and Dynamic Repartitioning
#
← Previous
5.7 Tensor Parallelism
→ Next
5.9 Sequence and Context Parallelism