Routing and Distributed Serving Scheduling

5.22.1Load Balancing, Join-Shortest-Queue, and Request Routing#

5.22.2KV-Aware, Prefix-Aware, and Cache-Aware Routing#

5.22.3Replica Placement and Session Affinity#

5.22.4Admission Control, Backpressure, and Rate Limiting#

5.22.5Autoscaling, Cold Start, and Warm Pools#

5.22.6Cross-Rack, Cross-Region, and Tiered Serving Architectures#