Transformer Structure and Its Evolution

6.4.1Encoder-Only, Decoder-Only, and Encoder–Decoder#

6.4.2Self-Attention, Cross-Attention, and FFN#

6.4.3Residual Connection, LayerNorm, and RMSNorm#

6.4.4Pre-Norm, Post-Norm, and Deep-Network Stability#

6.4.5GELU, GLU, GeGLU, and SwiGLU#

6.4.6Weight Tying, Embedding, and the LM Head#