AI Architecture & Systems
1
Microarchitecture
2
Kernel
3
Compiler
4
Architecture
5
System
6
Algorithms
System
/
§5.13
Training Data and Checkpoint Pipelines
5.13.1
Tokenization, Packing, and Dynamic Batching
#
5.13.2
Dataloader, Prefetch, and the CPU–GPU Pipeline
#
5.13.3
Distributed Data Sharding, Shuffling, and Recovery
#
5.13.4
Checkpoint Save/Load and Parallel I/O
#
5.13.5
Asynchronous Checkpointing and Incremental Saving
#
5.13.6
Consistency Among Data, Parameters, Optimizer, and Random State
#
← Previous
5.12 Training Memory Management
→ Next
5.14 Training Runtime and Numerical Stability