Collective Algorithms: Logical Communication Scheduling

SourcesSurvey §4.4: collective algorithms and their communication costs.

4.22.1Ring ReduceScatter/AllGather and Pipelining#

4.22.2Tree and Double Binary Tree#

4.22.3Recursive Halving/Doubling and Rabenseifner-Style AllReduce#

4.22.4Bruck-Style and Pairwise Exchange#

4.22.5Parallel Aggregated Trees (PAT)#

4.22.6Hierarchical Local–Global–Local Scheduling#

4.22.7TACCL and Topology-Specific Schedule Synthesis#

4.22.8Message Size, Rank Count, Startup Rounds, and Data Traffic#