Deploying and Tuning Quantized Models

5.24.1Precision Configuration for Weights, Activations, and KV Cache#

5.24.2Checkpoint Format, Packing, and Quantization Metadata#

5.24.3Matching Quantization Schemes, Kernels, and Hardware Support#

5.24.4Offline Quantization, Online Quantization, and Load-Time Conversion#

5.24.5Mixed-Precision Deployment for Dense/MoE/Multimodal Models#

5.24.6Joint Evaluation of Precision, Latency, Throughput, Capacity, and Cost#