Training Runtime and Numerical Stability

5.14.1Training Step, Autograd, and Optimizer Step#

5.14.2Mixed-Precision Training and the Scope of Hardware/Kernel Support#

5.14.3Loss Scaling, Gradient Clipping, and Anomaly Detection#

5.14.4Communication Precision, Accumulation Order, and Reproducibility#

5.14.5Diagnosing NaN/Inf, Loss Spikes, and Training Divergence#

5.14.6Co-Optimizing Operators, Compilers, and Distributed Training#