Distributed Runtime and Collective Communication

SourcesSurvey §4.4 for semantics and algorithms; corpus layer mappings for vendor runtime boundaries. Detailed API exercises remain extensions.

5.5.1Rank, World Size, Process Group, and Device Mesh#

5.5.2Collective Call Semantics, Completion Semantics, and Consistent Call Ordering#

5.5.3Broadcast, Reduce, AllReduce, AllGather, ReduceScatter, and All-to-All#

5.5.4NCCL/RCCL, Gloo/MPI/UCX, and Backend Selection#

5.5.5The Differing Boundaries of NCCom, ICI Runtime, and Compiler-Built-In Collectives#

5.5.6Topology Discovery, Algorithm Selection, and Communication Performance Tuning#

5.5.7Asynchronous Communication, Stream/Event, and Compute Overlap#

5.5.8Communication Deadlock, Timeout, Failure, and Diagnosis#