→ Back to Home
AWS

AWS Glue 6.0 Launches with 30% Cost Cut and Full Apache Iceberg v3 Support

AWS has officially released AWS Glue 6.0, introducing a major modernization of its serverless data integration service. The new release cuts operational pricing by 30% compared to previous versions and upgrades the underlying runtime to Apache Spark 4.1, Python 3.13, and Scala 2.13. Crucially, Glue 6.0 delivers full support for the Apache Iceberg v3 specification (built on Iceberg 1.11.0), headlined by the VARIANT data type with automatic shredding, nanosecond-precision timestamps, and native geospatial data types. Additionally, the update integrates Spark Declarative Pipelines for intent-driven ETL authoring, Arrow-native Python UDFs and UDTFs to eliminate JVM serialization bottlenecks, and a new real-time streaming mode engineered for single-digit millisecond latency. For data engineers and platform architects, this release resolves two of the most persistent operational headaches in data lakehouse maintenance: schema evolution friction and PySpark runtime serialization overhead. The addition of Iceberg v3 VARIANT shredding enables teams to query complex nested logs and JSON payloads with columnar efficiency without writing rigid extraction logic or maintaining redundant flattened copies. Furthermore, the 30% baseline price reduction immediately improves the total cost of ownership (TCO) for large-scale ETL workloads, providing FinOps leaders and engineering teams with tangible cost relief as data ingestion volumes continue to accelerate. The cloud data ecosystem is rapidly consolidating around open table formats, with Apache Iceberg establishing itself as the standard storage layer across hybrid and multi-engine architectures. Historically, proprietary cloud data warehouses held an efficiency edge over serverless lakehouse engines when parsing semi-structured data. By embedding Iceberg v3 VARIANT shredding alongside Spark 4.1's declarative execution directly into a fully managed serverless runtime, AWS narrows the capability gap between open data lakes and specialized proprietary warehouses, reinforcing the industry migration toward open, interoperable data architectures. Data practitioners managing existing AWS Glue workloads should evaluate upgrading to version 6.0 via AWS Glue Studio or using the automated upgrade agent to immediately capture the 30% cost savings and execution efficiencies. Teams ingesting high-velocity event streams or unstructured telemetry should pilot the VARIANT data type to phase out brittle JSON-flattening routines and simplify downstream table schemas. However, teams relying on custom Spark extensions or legacy Python dependencies must validate library compatibility against the Python 3.13 and Spark 4.1 runtimes before executing full-fleet production migrations.
#aws glue#apache iceberg#apache spark#data engineering#serverless
Read original source