Model Evolution: From Sequence Models to Foundation Models

6.27.1RNN, LSTM, and Seq2Seq#

6.27.2Attention and the Original Transformer#

6.27.3BERT, T5, and the Pretraining Paradigm#

6.27.4GPT, GPT-2, and GPT-3#

6.27.5InstructGPT and Instruction/Preference Alignment#

6.27.6Dense LM → MoE → Reasoning/Multimodal Models#