AI Architecture & Systems
1
Microarchitecture
2
Kernel
3
Compiler
4
Architecture
5
System
6
Algorithms
Kernel
/
§2.9
Exact Attention Kernels
2.9.1
Operator Decomposition and Memory Traffic of Standard Attention
#
2.9.2
FlashAttention: IO-Aware Tiling and Recomputation
#
2.9.3
FlashAttention-2/3 and Parallelism and Pipelining Optimizations
#
2.9.4
Causal Mask, Sliding Window, and Variable-Length Attention
#
2.9.5
Forward/Backward Attention Kernels
#
2.9.6
FlexAttention and Programmable Attention
#
← Previous
2.8 Fusion and Memory-Traffic Elimination
→ Next
2.10 Decode Attention and KV Cache Kernels