AI Architecture & Systems
1
Microarchitecture
2
Kernel
3
Compiler
4
Architecture
5
System
6
Algorithms
System
/
§5.7
Tensor Parallelism
5.7.1
Column-Parallel and Row-Parallel Linear Layers
#
5.7.2
Partitioning Attention Heads and MLPs
#
5.7.3
Vocabulary, Embedding, and Loss Parallelism
#
5.7.4
All-Reduce/All-Gather/Reduce-Scatter in TP
#
5.7.5
KV Head Count, Replication, and Partitioning Constraints
#
5.7.6
Single-Node and Cross-Node Tensor Parallelism
#
← Previous
5.6 Data Parallelism and Parameter Sharding
→ Next
5.8 Pipeline Parallelism