Tokenization and Input Representation

6.3.1BPE, WordPiece, Unigram, and Byte-Level Tokenization#

6.3.2Vocabulary, Embedding, and Output Projection#

6.3.3Special Tokens, Chat Templates, and Conversation Formats#

6.3.4Padding, Packing, Masking, and Sequence Boundaries#

6.3.5Text, Image, Audio, and Video Tokens#

6.3.6The Effect of Tokenization on Context Length and Compute Cost#