Future Directions and Architectural Evaluation

Designing a future AI datacenter is not just a matter of scaling today's accelerators, networks and racks. Every specialized architecture embodies assumptions about what workloads will look like and what they will demand of compute, memory and communication. Algorithms and hardware co-evolve — emerging models change the demands, and new hardware makes previously impractical algorithms economical — but they evolve on different timescales. Software and workloads shift during a deployment's lifetime, while architectural decisions stay fixed for far longer. Five challenges follow from that mismatch.

Workload specialization versus fleet flexibility. The workload mix can change while installed capacity stays the same. Pre-training holds optimizer state and issues large collectives; online inference is shaped by latency and throughput targets ^{[1, 2]}. Inference is not one profile either: mixture-of-experts models, multimodal inputs, long contexts, retrieval augmentation and diffusion all demand different balances of compute, memory capacity, bandwidth and communication ^[3]. The variation appears inside a single request — prefill offers more parallel work while decode spends more time moving weights and KV cache, and speculative decoding, batching, prefix caching, prefill–decode disaggregation, quantization and context length all shift the balance further. Google positions TPU 8t and TPU 8i for different workload mixes at the pool level ^[4], and OpenAI describes Jalapeño as specialized for inference yet balanced across a changing prefill/decode mix ^[5]. Separate pools improve efficiency but create a matching problem: one pool can sit idle while another queues.

Data placement and movement. Small-batch decoding can spend much of its time reading weights and KV cache ^[3]. Moving data closer to compute reduces transfers, but nearby capacity is limited and placement often needs explicit software management. Groq and Cerebras keep data in on-chip SRAM, with Cerebras extending it across a wafer ^{[6, 7]}, which avoids repeated device-DRAM traffic until parameter size, context length or concurrency exhausts the SRAM. Offloading adds transfers; distributing computation adds network communication. Under tensor parallelism weight shards stay put, but activations and partial results must still be exchanged. Processing-in-memory computes where operands live — d-Matrix Corsair inside SRAM arrays ^[8], Samsung HBM-PIM and SK hynix AiM inside DRAM devices ^{[9, 10]} — offering high internal bandwidth, though SRAM designs are capacity-limited and DRAM designs throughput-limited. Processing-near-memory puts arithmetic in adjacent logic, and 3D stacking shortens the link further: Raptor bonds logic directly to 3D DRAM to shrink the memory interface ^[11], gaining short paths without sacrificing throughput but introducing thermal and reliability challenges.

Communication across scale-up and scale-out. Constraints come from the interaction of workload mapping with physical design. Large gradient exchanges benefit from more link bandwidth because payload transfer dominates; tensor-parallel decoding with small batches instead exchanges short messages repeatedly at every layer and token ^[3]. Faster links shorten transmission but do not remove the cost of starting an exchange, traversing the network, or waiting on other devices — so the same fabric hits different limits as workload and batch size change. Topology helps partly: Trainium moves from torus to switched fabrics to cut path length and contention ^[12], and Boardfly cuts TPU 8i's worst-case path across ~1K chips from 16 hops to 7 ^[4] — but that is network diameter, and the effect on any collective still depends on its schedule and placement. Expanding a scale-up domain keeps more exchanges on the fast fabric at the cost of longer links; copper serves short reach and optics longer, and co-packaged optics shorten the electrical path between switch and optical interface without removing propagation delay or synchronization ^[13].

Power delivery and cooling from rack to facility. These can limit how much installed capacity is actually usable, as §4.8 works through in detail: adequate total power does not guarantee that each rack can receive it or dissipate the resulting heat.

Architectural specialization and model evolution. Architectures differ in how much of a model's computation is fixed in silicon versus left programmable, and that determines which model updates can be absorbed in software. SambaNova's SN40L compiler maps supported graphs onto configurable tiles and programs the routes, giving flexibility within a finite set of physical resources and supported operations ^[14]. Taalas HC1 goes the other way, hard-wiring the model while keeping configurable context lengths and low-rank adapters for fine-tuning ^[15]: updating an adapter changes behaviour, but replacing the hard-wired base model requires new silicon. Such chips can serve their original model as long as demand lasts, yet their capacity is hard to repurpose if demand moves to an unsupported model. Encoding model-specific function in hardware simplifies execution while making reuse depend on what stayed programmable. Across all these cases software can reassign work, remap computation or constrain activity — but the cost and effectiveness of those adaptations are set by the physical resources already deployed.