Automatic Differentiation and Training-Graph Compilation

3.4.1Reverse-Mode and Forward-Mode AD#

3.4.2Backward Graph, VJP, and JVP#

3.4.3AOT Autograd and Joint Graph Optimization#

3.4.4Activation Liveness and Rematerialization#

3.4.5Gradient Accumulation, Gradient Communication, and the Optimizer Graph#