Sparse and Compressed Attention

6.8.1Token, Block, Head, and Layer-Level Sparsity#

6.8.2Local, Global, Dilated, and Sliding-Window Attention#

6.8.3Longformer, BigBird, and Sparse Connectivity Patterns#

6.8.4Native Sparse Attention (NSA)#

6.8.5Mixture of Block Attention (MoBA)#

6.8.6DeepSeek Sparse Attention (DSA)#

6.8.7Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA)#

6.8.8Dynamic Token Selection, Top-k, and Trainable Sparsity#