Batching and Online Scheduling

5.18.1Static Batching, Dynamic Batching, and Continuous Batching#

5.18.2Iteration-Level Scheduling and Token Budgets#

5.18.3Chunked Prefill and Decode Priority#

5.18.4Preemption, Recompute, and Swap#

5.18.5CPU–GPU Overlap and Low-Overhead Schedulers#

5.18.6Fairness, Priority, and Mixing Short and Long Requests#

5.18.7Load-Aware Batch Size and Adaptive Scheduling#