SageMaker AI Adds EC2 G6e Instances to Batch Transform for Scalable Offline Inference

AWS has announced support for Amazon EC2 G6e instances within Amazon SageMaker AI Batch Transform. EC2 G6e instances are equipped with up to eight NVIDIA L40S Tensor Core GPUs—offering 48 GB of memory per GPU, reaching 384 GB across a fully configured node—paired with third-generation AMD EPYC processors. This integration enables practitioners to target GPU-intensive offline batch inference workloads directly against datasets stored in Amazon S3 without requiring persistent, always-on inference endpoints. As generative AI pipelines transition from interactive demos to automated enterprise back-office workflows—such as re-indexing massive vector stores, running offline model evaluation suites, or transcribing media repositories—hosting models on continuous endpoints introduces substantial cost and operational inefficiencies. G6e instances bridge the price-performance divide between general-purpose inference hardware and ultra-expensive distributed training clusters. For DevOps engineers and ML platform architects, the NVIDIA L40S hardware brings optimized FP8/FP16 tensor throughput alongside expansive GPU memory capacity, effectively eliminating out-of-memory bottlenecks when processing large-batch transformer workloads asynchronously. This release reflects the broader maturation of cloud AI infrastructure toward workload-segmented compute tiers. With AI inference spending rapidly expanding across enterprise architectures, hyperscalers and platform teams are prioritizing cost predictability and hardware utilization efficiency over monolithic cluster provisioning. By combining dense, single-node multi-GPU compute with SageMaker's managed batch execution layer, AWS is catering to organizations seeking to optimize token economics across non-real-time data pipelines without managing low-level container orchestration or GPU scheduling. In practice, infrastructure teams operating large-scale offline AI jobs should evaluate migrating legacy G5-based or unmanaged GPU fleets to G6e Batch Transform configurations, particularly for memory-intensive models and quantized architectures. Because SageMaker AI automatically spins up the G6e compute, pulls input batches from Amazon S3, writes predictions back to cloud storage, and tears down the infrastructure immediately upon completion, teams eliminate idle capacity costs entirely. Practitioners should profile data ingestion rates to ensure storage read/write pipelines remain saturated, preventing I/O bottlenecks when utilizing all eight L40S accelerators concurrently.