Observability, Benchmarking, and Troubleshooting

5.39.1Metrics, Logs, Tracing, and Cross-Component Timelines#

5.39.2PyTorch Profiler, Nsight, ROCm Tools, and Device-Specific Profilers#

5.39.3Localizing Issues Across Framework/Compiler/Kernel/Runtime/Driver#

5.39.4Joint Monitoring of GPU/NPU/CPU/Network/Storage#

5.39.5Offline/Online Benchmarking and Real Traffic Replay#

5.39.6NCCL Hangs, OOM, Memory Leaks, and CPU Bottlenecks#

5.39.7Performance Regressions, Version Pinning, Numerical Consistency, and Experiment Reproducibility#

5.39.8MLPerf, Serving Benchmarks, and Reporting Conventions#