Hybrid Parallelism and Automatic Planning

SourcesSurvey §4.4 and §6.3 motivate communication placement; distributed-training implementation topics are retained.

5.11.1Combining DP/TP/PP/SP/CP/EP#

5.11.2Device Mesh, Parallel Groups, and Topological Layering#

5.11.3Model Partitioning, Resharding, and Layout Conversion#

5.11.4Memory Constraints, Message Size, Communication Latency, and Parallelism Choice#

5.11.5Communication Placement Across Scale-Up/Scale-Out Domains#

5.11.6Heterogeneous Devices, Stragglers, and Uneven Sharding#

5.11.7Automatic Parallelization, Search, and Performance Models#