Data, Training Recipes, and Model Scaling

6.12.1Data Collection, Filtering, Deduplication, and Quality Assessment#

6.12.2Data Mixture, Curriculum, and Multilingual Training#

6.12.3Synthetic Data, Self-Training, and Data Distillation#

6.12.4Model Width, Depth, Vocabulary, and Sequence Length#

6.12.5Continual Learning, Domain Adaptation, and Catastrophic Forgetting#

6.12.6Data Quality, Token Budget, and Scaling Trade-offs#