Benchmarking TPU v6e: Gemma 3 Workload Profiles Reveal Critical Prefill vs Decode Divergence

Google Cloud published a detailed benchmarking study evaluating Gemma 3 12B and Gemma 3 27B models on Google Cloud TPU v6e accelerators orchestrated via Google Kubernetes Engine (GKE) Autopilot and served through vLLM. The benchmark evaluated distinct request profiles—specifically prefill-heavy classification (4,000 input tokens, 10 output tokens) against decode-heavy generation—across concurrency levels scaling from 16 to 128 concurrent users. The results demonstrated that while both 12B and 27B models achieve equivalent linear throughput scaling (between 6.0x and 6.4x at 128 users) on classification tasks, generation tasks exhibit a sharp divergence: Gemma 3 12B scaled to an 8.19x relative throughput multiplier, whereas the 27B model hit a rigid saturation ceiling at 4.12x past 64 concurrent users. This divergence fundamentally challenges the standard practice of using general-purpose hardware benchmarks or raw parameter volume to estimate cloud inference capacity. For MLOps engineers and platform teams, sizing TPU clusters based on generic token-per-second benchmarks leads either to under-provisioned clusters suffering from request timeouts or drastically over-provisioned infrastructure that inflates operational expenditure. Organizations running document classification, moderation, or compliance checks can safely deploy higher-parameter models like Gemma 3 27B without incurring a concurrency penalty. Conversely, teams running open-ended text generation, conversational agents, or agentic loop reasoning must recognize that memory bandwidth constraints during the auto-regressive decode phase create strict concurrency walls. The findings reflect a broader architectural shift across the AI infrastructure ecosystem, where the decoupling of prefill and decode phases is becoming critical for cost-effective inference serving. As models scale and enterprise adoption transitions from pilot prototypes to high-traffic production APIs, inference bottlenecks increasingly stem from memory bandwidth limitations and key-value (KV) cache overhead rather than pure matrix multiplication capacity. Serving engines like vLLM running on custom silicon topologies (such as TPU v6e 2x2 chip slices) require fine-grained runtime tuning to avoid memory fragmentation and zero-padding compute waste during dynamic batching. In practice, infrastructure architects should avoid relying on traditional CPU, memory, or basic GPU/TPU utilization metrics for autoscaling inference replicas. Because hardware saturation manifests as severe end-to-end (E2E) latency spikes and silent request dropouts before raw compute hits 100%, Horizontal Pod Autoscalers (HPA) in GKE should be keyed directly to E2E latency percentiles and active queue depth. Furthermore, practitioners deploying on TPUs with vLLM must configure linear sequence bucket padding using parameters like VLLM_TPU_BUCKET_PADDING_GAP rather than default geometric padding, preventing massive memory allocations on long input contexts. For high-concurrency generative tasks, teams should either cap concurrent requests at 64 per 27B replica or deploy the 12B variant to preserve linear scaling characteristics.