Generational Evolution and Cross-Generation Comparison

Tracing one accelerator family across generations separates the features that persist from those that change — a distinction the architectural taxonomy alone cannot make.

The divergence in that figure is the central fact of the last decade of GPU design. General-purpose high-precision SIMD throughput (FP32) has grown slowly and remains below 200 TFLOPS, while low-precision matrix-engine throughput has risen to roughly 2,500 TFLOPS. Each generation's headline gain comes from the matrix engine and from ever-lower precision, not from the general-purpose datapath.

Raising matrix throughput, though, is only useful if operands arrive fast enough, so the same generations also reworked data supply and cooperation scope. NVIDIA's Tensor Core evolution includes redesigned operand-delivery mechanisms and tighter coordination among execution units, enabling greater overlap between data movement and computation and supporting larger matrix operations ^{[1, 2]}. Scheduling mechanisms, data-movement support and on-chip SRAM capacity all advanced alongside the arithmetic; the microarchitectural detail of that progression, from Volta through Blackwell, is traced in §1.8.

Peak-throughput numbers are only comparable if the conventions behind them are stated, so this comparison fixes them first: the values below are vendor-reported dense peaks in decimal units, FP32/FP64 use the listed scalar or vector peaks, lower-precision columns use matrix- or tensor-engine throughput where available, and a fused multiply-add counts as two operations. A dash means the cited source does not specify the format or the hardware does not support it. Years are the first public family disclosure; TPU and Neuron FP16/BF16 values are BF16.

P100 and V100 entries do not imply BF16 support. MI100 reaches 184.6 TFLOPS in FP16 but 92.3 TFLOPS in BF16. Google's reported FP8 peaks for TPU v5p and v6e equal their BF16 peaks and should not be read as a native FP8 throughput increase. B300's dense INT8 value is half the cited 330 sparse TOPS, and Trainium 3 processes MXFP4 through its MXFP8 path.

Read down the GPU columns and the pattern from §4.7 repeats numerically: FP64 and FP32 creep upward while FP16 matrix throughput multiplies and entirely new, lower-precision formats — FP8, then FP4 — appear and immediately dominate the headline figures. The specialized path grows; the general-purpose one does not.

Structured sparsity is the second lever. Trained networks hold substantial redundancy, so many weights can be zeroed after pruning with little accuracy loss, cutting both computation and memory traffic. Fully unstructured sparsity is impractical in hardware because of irregular access and metadata overhead, so accelerators enforce an M:N rule — only M of every N values nonzero — which preserves locality, allows compact storage, and keeps the datapath regular enough for a matrix engine ^[3]. NVIDIA and AMD GPUs implement a fixed 2:4 pattern with lightweight metadata, roughly doubling throughput when the constraint holds ^{[4, 5, 6]}; AWS Neuron generalizes to 4:16, 4:12, 4:8, 2:8, 2:4, 1:4 and 1:2 for finer accuracy–throughput control ^[7]. The bit-level layout of these formats and the 2:4 representation itself are covered in §1.2.

For GPUs, on-chip SRAM here denotes L2/shared cache; for NPUs it denotes a disclosed software-managed scratchpad. Interconnect values are aggregate bidirectional bandwidth per accelerator. TPU 8's 19.2 Tb/s ICI is expressed as 2,400 GB/s.

From the earliest to the latest listed NVIDIA and AMD generations, HBM capacity grew roughly 13.5–18× and HBM bandwidth roughly 23–27×. The two scale for largely independent reasons. Capacity improves through higher per-stack density — more dies per stack and denser DRAM dies — and through more stacks per package. Bandwidth improves through higher per-pin signalling rates across HBM generations and through wider aggregate I/O, mostly from additional stacks. Chiplet-based packaging supports both by enlarging the effective integration area, making room for more stacks with their PHYs and controllers.

Because memory has lagged compute, accelerators lean harder on larger on-chip SRAM. Through 2025 shared-cache capacity grew about 32× for NVIDIA and 64× for AMD relative to the earliest listed devices — "shared cache" meaning NVIDIA's L2 and AMD's large last-level cache. Bigger caches improve locality, amplify reuse and relieve off-chip bandwidth.

The retained chart shows endpoint growth of roughly 120× and 96× for throughput, 32× and 64× for shared cache, and 18× and 9× for HBM capacity. These plotted values predate the updated table specifications, but the qualitative result remains: compute growth outruns capacity growth for both.

On the fabric side, interconnect bandwidth has risen steadily while topologies have changed shape. AWS Trainium illustrates the trade cleanly ^{[8, 9, 10]}: Trn1 and Trn2 connect 16 chips in a 4×4 2D torus, Trn2 UltraServer links four such systems into a 64-chip domain through inter-instance rings, and Trainium 3 replaces the torus with the switched all-to-all NeuronSwitch-v1 fabric supporting up to 144 chips. A low-radix torus avoids switch silicon and scales incrementally, but diameter and contention grow with size; a switched fabric costs ports, area and power but shortens paths and makes performance less sensitive to placement — which matters most for communication-intensive collectives such as mixture-of-experts routing.

Switching is not the only alternative to a torus. Google TPU 8i's Boardfly builds a hierarchical high-radix network from four-chip blocks: eight boards form a fully connected group, and 36 groups are joined through optical circuit switches, giving 1,152 physical chip positions and up to 1,024 active chips. For the 1,024-chip configuration Google reports a maximum path of seven hops against 16 for the referenced 3D torus, a 56% reduction ^[11]. Shorter paths matter most for latency-sensitive collectives: during autoregressive decoding, small reductions across model-parallel shards can be a substantial fraction of per-token execution time.