The Full Lifecycle of an Inference Request

5.17.1API Server, Tokenizer, and Request Queue#

5.17.2Scheduler, Worker, Model Runner, and Executor#

5.17.3Model Configuration, Weight Loading, and Parameter Initialization#

5.17.4Prefill, Decode, Sampling, and Detokenization#

5.17.5Streaming Output, Cancellation, and Resource Reclamation#

5.17.6Embedding, Reranking, Reward, and Generation Requests#