Linear Attention, SSMs, and Hybrid Models

6.9.1Kernelized Linear Attention and Linear Complexity#

6.9.2RetNet, RWKV, and Recurrent State#

6.9.3S4, Mamba, and Mamba-2#

6.9.4DeltaNet and Gated DeltaNet#

6.9.5Attention–SSM/Linear-Attention Hybrid Architectures#

6.9.6Prefill Scan, Recurrent Decode, and State Capacity#