Context, Scope, and Evidence Standards

Progress in AI has stopped being only an algorithms story. Nearly 90% of notable AI models in 2024 came from industry, up from 60% in 2023, and the training compute behind frontier models has been doubling roughly every five months ^[1]; one estimate puts the growth at about 5× per year since 2020 ^[2]. Capability is now coupled to whether an organization can provision and operate large infrastructure, not to single-processor speed.

That demand has made energy a first-order architectural constraint rather than a facilities detail. The International Energy Agency puts global datacenter electricity use at roughly 415 TWh in 2024 and projects it could more than double to about 945 TWh by 2030, with AI a major driver ^[3]. In the United States, Lawrence Berkeley National Laboratory reports datacenters at about 4.4% of total electricity consumption in 2023, projected to reach 6.7–12% by 2028 ^[4].

The deeper problem is not that any one resource is scarce — it is that the resources are growing at very different rates.

Across the selected observations, total model parameter counts grow at an endpoint annualized rate of roughly 248% per year ^{[5, 6]}. Per-accelerator dense FP16/BF16 compute follows at about 67%. HBM bandwidth and capacity trail at roughly 47% and 42%, and scale-up interconnect bandwidth is slowest at about 30%. The comparison is directional rather than a claim that every model or device generation follows the same curve: model size can outpace compute, while memory and communication improve more slowly than arithmetic throughput. Independent analyses of the "memory wall" report the same pressure ^[7].

This is why an accelerator cannot be judged by its arithmetic throughput. Four pressures — compute, memory capacity, memory and interconnect bandwidth, and power — tighten at different speeds, and a design that relieves one often intensifies another. Specialized matrix engines and low-precision arithmetic raise compute efficiency; higher memory bandwidth and greater on-chip reuse cut trips to external memory; fast interconnects let model state span many devices. Because each mechanism targets a different bottleneck, architectures differ mainly in which pressure they chose to relieve first.

Alongside continued GPU development, cloud and platform providers began building accelerators for the services they operate. Google disclosed its TPU in 2016 ^[8], AWS announced Inferentia in 2018 ^[9], and Meta and Microsoft introduced MTIA and Maia in 2023 ^{[10, 11]}.

These accelerators encode different objectives. NVIDIA and AMD datacenter GPUs are programmable platforms spanning a broad range of training and inference work, pairing specialized matrix computation with general-purpose parallel execution ^{[12, 13]}. In-house accelerators are usually tailored to their owner's own models, service requirements and cost structure. AWS built Inferentia to cut inference cost at high throughput and low latency, then Trainium for training ^{[9, 14]}. Google's first TPU targeted inference, with v2 and v3 extending the family to training ^[15]; more recently TPU 8t emphasizes large-scale pre-training while TPU 8i targets post-training and latency-sensitive serving ^[16].

Workloads shift across generations too. Meta's MTIA 100 and 200 focused on ranking and recommendation inference ^{[10, 17]}; MTIA 300 extends to recommendation training and MTIA 400 broadens to generative AI while keeping recommendation ^[18]. Microsoft's Maia 100 arrived for cloud AI training and inference, whereas Maia 200 targets the cost of token generation during inference ^{[11, 19]}.