AWS Eliminates Local Broker Storage Overhead with S3 Files for Diskless Apache Kafka

AWS published an architectural design and benchmark analysis showing how organizations can operate Apache Kafka in a completely diskless model using Amazon S3 Files. By mounting an S3-backed shared file system over NFS and configuring Kafka log directories directly against the mount point, Kafka brokers read and write to Amazon S3 as if it were a local POSIX-compliant file system. Active stream segments reside in S3 Files' low-latency caching tier, while older segments remain durably persisted across S3's eleven nines of object storage durability. This pattern tackles one of Kafka's most persistent operational bottlenecks: disk provisioning and storage sprawl. In traditional Kafka architectures, brokers require dedicated block storage (like Amazon EBS) that must be continuously right-sized to prevent disks from filling up during consumer lag or retention spikes. Platform teams frequently maintain fragile custom tooling or specialized tiered storage plugins to archive segments into object storage and reclaim local block space. By delegating storage retention directly to S3 Files, compute instances can be treated as truly stateless, allowing brokers to fail, restart, or scale horizontally without lengthy partition rebalancing or volume recovery procedures. This architecture reflects a wider industry shift toward blurring the boundary between POSIX file interfaces and scalable object storage substrates. As AI training and streaming data engines mature, maintaining distinct storage protocols for streaming, analytics, and archiving introduces unnecessary data movement and operational toil. Exposing S3 buckets through high-throughput, low-latency file system semantics allows legacy file-based tools, machine learning pipelines, and distributed streaming clusters to access unified object storage without SDK overhead or rewriting core engine storage engines. In practice, engineers evaluating diskless Kafka on S3 Files should carefully model their read and write traffic profiles against S3 Files caching thresholds. While recent active segments achieve millisecond latency from the local cache, historical replay requests that miss the cache will trigger transparent object fetches from S3, potentially impacting tail read latency for lagging consumers. Teams running Kafka on EC2 or self-managed Kubernetes should benchmark their maximum sustained throughput against network I/O limits before phasing out block storage entirely.