GKE Inference Gateway Accelerates AI Inference with Prefix Caching

Google Cloud has unveiled the GKE Inference Gateway, a new feature designed to significantly boost the performance and efficiency of AI inference workloads running on Google Kubernetes Engine. This innovation addresses the growing demand for optimized infrastructure as generative AI models transition from experimental stages to large-scale production environments. The GKE Inference Gateway distinguishes itself from traditional load balancing methods, which often rely on simple round-robin distribution. Such conventional approaches can lead to frequent and expensive recomputations on accelerators and increased user latency. In contrast, the GKE Inference Gateway employs sophisticated capabilities like prefix caching and model-aware routing. This intelligent system ensures that incoming requests are directed to the most appropriate accelerator that is already primed to process them, thereby maximizing hardware utilization and minimizing idle time. An independent benchmark report by Principled Technologies has validated these architectural advantages. The report compared GKE, equipped with the Inference Gateway, against a standard third-party managed Kubernetes service using traditional HTTP load balancing. Testing was conducted on a Llama 3.1 8B Instruct shared prefix workload, utilizing identical hardware configurations of eight NVIDIA A100 40GB GPUs. The results demonstrated a substantial performance gap. GKE with the Inference Gateway achieved 15.7% higher throughput, meaning it processed more tokens per second and could handle higher request capacities or reduce hardware requirements for the same workload. Crucially, it delivered a 92.8% shorter time to first token (TTFT), leading to significantly quicker perceived response starts for interactive AI applications. Furthermore, inter-token latency was reduced by 62.6%. These improvements are pivotal for transforming LLM-based applications from being sluggish and expensive to fast and production-grade, offering a compelling solution for organizations looking to self-host models efficiently.