Collective Semantics: Endpoint Data Transformation

SourcesSurvey §4.4 and Fig. 9: collective endpoint semantics.

4.21.1AllReduce: Aggregation and Full-Result Replication#

4.21.2AllGather: Shard Collection and Concatenation#

4.21.3ReduceScatter: Aggregation with Retained Result Shards#

4.21.4All-to-All: Destination-Rank-Oriented Data Exchange#

4.21.5Implementing AllReduce as ReduceScatter + AllGather#

4.21.6Communication Requirements of DDP, FSDP/ZeRO, TP/SP, and EP#

4.21.7Distinguishing Collective Operation, Algorithm, and Physical Topology#