Sparsification, Pruning, and Model Compression

6.21.1Unstructured, Structured, and N:M Pruning#

6.21.2Magnitude, Gradient, and Second-Order Pruning#

6.21.3SparseGPT, Wanda, and Post-Training Pruning#

6.21.4Activation Sparsity, Token Pruning, and Early Exit#

6.21.5Head, Layer, and Expert Pruning#

6.21.6Low-Rank Factorization and Structural Compression#

6.21.7Knowledge Distillation, Self-Distillation, and Teacher–Student Learning#