Transformer Attention Mechanisms: Scaled Dot-Product vs Multi-Query Attention (MQA)

The Memory Bandwidth Bottleneck

Standard Multi-Head Attention (MHA) maintains $H$ distinct Key and Value heads. Grouped-Query Attention (GQA) shares KV heads across query head groups, yielding 8x memory bandwidth reduction during generation.

References & Further Reading:

• GQA: Training Generalized Multi-Query Transformer Models (arXiv:2305.13245)