Pretraining Objectives and Optimization Algorithms

6.11.1Causal LM, Masked LM, and Denoising Objectives#

6.11.2Next-Token Prediction and Multi-Token Prediction#

6.11.3SGD, AdamW, Adafactor, and Muon#

6.11.4Learning Rate Schedule, Warm-up, and Weight Decay#

6.11.5Gradient Clipping, Batch Scaling, and Optimization Stability#

6.11.6Low-Precision Training, Error Accumulation, and Precision Sensitivity#