Multimodal Foundation Models

6.22.1Vision Encoder, Projector, and Language Backbone#

6.22.2CLIP/SigLIP, ViT, and Contrastive Learning#

6.22.3LLaVA-Style Alignment and Visual Instruction Tuning#

6.22.4Early Fusion, Late Fusion, and Native Multimodality#

6.22.5Cross-Attention, Image Tokens, and Dynamic Resolution#

6.22.6Audio/Video Tokenization and Cross-Modal Temporal Alignment#

6.22.7ASR, TTS, Codec LMs, and Omni Models#