Google Details TPU 8t and 8i Architecture to Decouple Training from Agentic Inference

At the Hot Chips symposium, technical architectural details emerged for Google's eighth-generation custom machine learning silicon, featuring a formal bifurcation into two purpose-built processors: the TPU 8t and the TPU 8i. The TPU 8t is engineered specifically for dense model training and large cluster scalability, powered by a dedicated two-layer switching topology named the Virgo network fabric that links up to 134,000 TPUs with 47 petabits per second of bisectional bandwidth. Concurrently, the TPU 8i targets high-throughput, low-latency inference and agentic decoding. Hosted natively on Google's Arm-based Axion CPUs at a 2:1 ratio and incorporating in-field unit testing for heightened reliability, availability, and serviceability (RAS), the eighth-generation TPU system redesigns how custom silicon interfaces with cloud compute and network infrastructure. For machine learning engineers and cloud architects, this hardware divergence solves a critical efficiency mismatch in hyperscale infrastructure. Running auto-regressive decoding, agentic reasoning chains, and Mixture-of-Experts (MoE) routing on monolithic accelerators optimized for matrix-heavy pre-training leads to underutilized compute units and spiraling cloud costs. By isolating training-specific hardware features from inference-specific needs—such as large on-chip SRAM allocations, specialized collective communication engines, and host-level Arm processing—teams deploying LLMs can achieve significantly higher token throughput and lower latency without paying the operational power penalty of general-purpose clusters. This transition reflects a fundamental shift across the AI hardware industry away from universal compute blocks toward workload-specialized silicon fabrics. As leading frontier AI developers scale model deployments across millions of concurrent users, memory bandwidth and inter-chip synchronization have replaced raw floating-point operations (FLOPs) as the primary execution bottlenecks. Google's integration of Axion CPUs with TPU 8i mirrors broader hyperscaler movements toward end-to-end custom silicon stacks, challenging merchant GPU architectures by co-designing the entire execution chain—from the host processor and interconnect to bare-metal compilers like JAX, PyTorch (TorchTPU), and open serving engines like vLLM. For platform teams and ML practitioners, the bifurcated accelerator design requires rethinking workload placement and cluster provisioning. Infrastructure engineers should prepare to partition training pipelines and real-time serving clusters onto dedicated silicon tiers rather than provisioning homogenous GPU pools. DevOps teams running on Kubernetes (GKE) should evaluate workload-tailored orchestration, accounting for non-virtualized bare-metal hardware access, Axion host architectures, and frameworks that natively support custom collective engines to maximize tokens per dollar and prevent memory saturation in production.