Serving Frameworks, Component Ecosystem, and Device Backends

SourcesOriginal serving outline; corpus mappings motivate backend-specific coverage checks rather than assuming every framework runs on every accelerator.

5.28.1vLLM: Engine, Scheduler, Worker, and PagedAttention#

5.28.2SGLang: Scheduler, RadixAttention, and Model Runner#

5.28.3TensorRT-LLM and NVIDIA Triton Inference Server#

5.28.4NVIDIA Dynamo, llm-d, and Distributed Inference Orchestration#

5.28.5Ray Serve and Multi-Stage Serving Pipelines#

5.28.6llama.cpp, MLX, and CPU/Local Inference#

5.28.7FlashInfer, NIXL, and Reusable Inference Components#

5.28.8The Backend Compatibility Matrix: Model, Precision, Kernel, Device, and SDK Version#

5.28.9The Boundary Between Vendor SDK/Serving Entry Points and General Serving Frameworks#