MHA, MQA, GQA, and MLA

6.6.1Multi-Head Attention (MHA)#

6.6.2Multi-Query Attention (MQA)#

6.6.3Grouped-Query Attention (GQA)#

6.6.4Multi-Head Latent Attention (MLA)#

6.6.5Head Sharing, Latent Compression, and Cache Capacity#

6.6.6KV Cache and Incremental Autoregressive Inference#

6.6.7Quality, Bandwidth, and Parallelism Trade-offs Among Attention Variants#