Non-GEMM Operators and Dedicated Support

1.16.1Softmax, LayerNorm, and RMSNorm#

1.16.2Activation Functions, Exponential, Reciprocal, and Square Root#

1.16.3Reduction, Scan, Sort, and Top-k#

1.16.4Embedding, Gather/Scatter, and Sparse Memory Access#

1.16.5Hardware Requirements of Attention and MoE#

1.16.6Compute-Unit Balance and Amdahl's Law#