The Basic Mechanism of Attention

6.5.1Scaled Dot-Product Attention#

6.5.2Q, K, V, and Attention Scores#

6.5.3Causal, Bidirectional, and Cross-Attention Masks#

6.5.4Softmax, Normalization, and Stability#

6.5.5Compute, Memory, and Long-Sequence Complexity of Attention#

6.5.6The Boundary Between Exact Attention and Sparse/Approximate Attention#