Introducing Flink's Native S3 FileSystem: Built for Performance, Designed for Production

Apache Flink has introduced a significant advancement in its interaction with Amazon S3 object storage through the release of `flink-s3-fs-native`, a new, purpose-built S3 filesystem. This development aims to address the limitations and complexities associated with previous Hadoop-based S3 plugins that Flink relied upon for data operations, checkpointing, and savepoints. The older plugins, such as `flink-s3-fs-hadoop` and `flink-s3-fs-presto`, often forced users to compromise between critical functionalities like exactly-once semantics for data sinks and efficient read performance. The new `flink-s3-fs-native` is a ground-up, Hadoop-free implementation, shipping as an experimental opt-in plugin with Flink 2.3. It has already been validated in production environments at major technology companies, demonstrating measurable and reproducible performance improvements. A key feature of this native filesystem is its support for exactly-once recoverable writes, leveraging S3 multipart uploads. This ensures data consistency and fault tolerance, allowing uploads to be resumed seamlessly in case of failures. Furthermore, the plugin introduces a clean and unified configuration model, enabling per-bucket configuration for various S3 environments. This means a single Flink cluster can interact with multiple S3 buckets, each with distinct credentials, regions, endpoints, and encryption policies, without the complexity of Hadoop-style key mirroring or adapter translation layers. This streamlined configuration is particularly beneficial for platform and infrastructure teams managing multi-tenant Flink clusters. The deprecation of the AWS SDK for Java 1.x, which reached its end-of-support in December 2025, also underscores the necessity of this new native plugin. By removing the dependency on outdated SDKs and the extensive Hadoop common dependency tree, `flink-s3-fs-native` offers a more modern, lightweight, and maintainable solution for Flink's S3 integration. The Flink community encourages users to deploy and test this new plugin to provide feedback, helping to mature it from an experimental feature to the default S3 filesystem for Flink.