AI Architecture & Systems
1
Microarchitecture
2
Kernel
3
Compiler
4
Architecture
5
System
6
Algorithms
System
/
§5.12
Training Memory Management
5.12.1
Activation Checkpointing and Selective Recomputation
#
5.12.2
CPU/NVMe Offload and State Tiering
#
5.12.3
Activation Offload, Prefetch, and Asynchronous Transfer
#
5.12.4
Optimizer State Sharding and Low-Precision States
#
5.12.5
Memory Budget, Peak Memory, and Fragmentation
#
5.12.6
Training Graphs, Communication Buffers, and CUDA Graph Memory
#
← Previous
5.11 Hybrid Parallelism and Automatic Planning
→ Next
5.13 Training Data and Checkpoint Pipelines