AI Architecture & Systems
1
Microarchitecture
2
Kernel
3
Compiler
4
Architecture
5
System
6
Algorithms
Algorithms
/
§6.26
Communication-Efficient Learning and Distributed Optimization
6.26.1
Local SGD and Periodic Parameter Averaging
#
6.26.2
Gradient Quantization, Sparsification, and Error Feedback
#
6.26.3
Low-Rank Gradient Compression and PowerSGD
#
6.26.4
Asynchronous Training, Staleness, and Convergence
#
6.26.5
Low-Bandwidth Distributed Pretraining and DiLoCo
#
6.26.6
Federated Learning, Secure Aggregation, and Differential Privacy
#
6.26.7
Communication Budget, Statistical Efficiency, and Wall-Clock Training Time
#
← Previous
6.25 AI Workloads Beyond LLMs
→ Next
6.27 Model Evolution: From Sequence Models to Foundation Models