Autoregressive Inference and Sampling

6.16.1Prefill, Decode, and Token-by-Token Generation#

6.16.2Greedy, Temperature, Top-k, and Top-p#

6.16.3Beam Search, Length Penalty, and Repetition Control#

6.16.4Logits Processor, Stopping Criteria, and Control Tokens#

6.16.5Constrained Decoding, Grammars, and Structured Output#

6.16.6Generation Quality, Diversity, Latency, and Reproducibility#