Sequence and Context Parallelism

5.9.1The Boundary Between Sequence Parallelism and Context Parallelism#

5.9.2Sequence Partitioning for Long-Context Attention#

5.9.3Ring Attention and KV Block Passing#

5.9.4Ulysses and All-to-All Sequence Redistribution#

5.9.5Hierarchical, Hybrid, and Two-Dimensional Context Parallelism#

5.9.6Causal Masking, Load Balance, and Communication Hiding#