AI Architecture & Systems
1
Microarchitecture
2
Kernel
3
Compiler
4
Architecture
5
System
6
Algorithms
System
/
§5.10
Expert Parallelism
5.10.1
Expert Placement, Replication, and Sharding
#
5.10.2
Token Dispatch, All-to-All, and Combine
#
5.10.3
Combining Expert Parallelism with Tensor Parallelism
#
5.10.4
Expert Load Balance and Hot Experts
#
5.10.5
Dropless Routing and Capacity Management
#
5.10.6
Cross-Node MoE Communication and Topology-Aware Deployment
#
← Previous
5.9 Sequence and Context Parallelism
→ Next
5.11 Hybrid Parallelism and Automatic Planning