Position Encoding and Context Extension

6.7.1Absolute and Relative Position Embeddings#

6.7.2RoPE, ALiBi, and Position Representation#

6.7.3RoPE Scaling, Position Interpolation, and YaRN#

6.7.4Sliding Window, Attention Sink, and Streaming Context#

6.7.5Long-Context Training, Extrapolation, and Retrieval Ability#

6.7.6Context Compression, Memory, and Cross-Segment State#