6. AlgorithmsModels, Training Methods, and Generation Algorithms

Retained model and algorithm curriculum. The hardware survey motivates workload diversity and adaptability, but is not the source for the detailed algorithms or model-family histories below.

6.1Deep Learning and Model Computation Fundamentals6 6.2The Full Lifecycle of a Foundation Model6 6.3Tokenization and Input Representation6 6.4Transformer Structure and Its Evolution6 6.5The Basic Mechanism of Attention6 6.6MHA, MQA, GQA, and MLA7 6.7Position Encoding and Context Extension6 6.8Sparse and Compressed Attention8 6.9Linear Attention, SSMs, and Hybrid Models6 6.10Mixture of Experts7 6.11Pretraining Objectives and Optimization Algorithms6 6.12Data, Training Recipes, and Model Scaling6 6.13SFT and Parameter-Efficient Fine-Tuning6 6.14Alignment and Preference Learning6 6.15Reasoning and RL Post-Training7 6.16Autoregressive Inference and Sampling6 6.17Test-Time Compute and Inference Strategies6 6.18Speculative Decoding10 6.19Parallel, Blockwise, and Diffusion Language Generation7 6.20Quantization: Algorithms and Numerical Methods8 6.21Sparsification, Pruning, and Model Compression7 6.22Multimodal Foundation Models7 6.23Diffusion, Flow Matching, and Visual Generation8 6.24Retrieval, Tools, and Agent Algorithms7 6.25AI Workloads Beyond LLMs7 6.26Communication-Efficient Learning and Distributed Optimization7 6.27Model Evolution: From Sequence Models to Foundation Models6 6.28Model Family: Llama6 6.29Model Family: Mistral and Mixtral5 6.30Model Family: DeepSeek7 6.31Model Family: Qwen6 6.32Model Family: Kimi6 6.33Model Family: GLM5 6.34Other Model Families and Reading Public Reports6 6.35Model Evaluation and Cross-Layer Trade-offs8