Meta's AI Storage Blueprint at Scale

Meta has unveiled insights into its advanced storage infrastructure, specifically engineered to address the escalating demands of artificial intelligence at an unprecedented scale. The company notes a significant disparity between the rapid advancements in AI compute capabilities and the comparatively slower progress in storage and interconnect performance. This imbalance frequently leads to storage bottlenecks, which directly hinder GPU efficiency and prolong the time required for AI research and development. These storage limitations are a primary contributor to GPU stalls for AI workloads, impacting both expenditures and time-to-market for new AI innovations. To overcome these challenges, Meta employs a sophisticated, multi-tiered storage strategy. At its core is Tectonic, a regional, multi-tenant foundational block layer. Tectonic is designed for extreme durability and availability, leveraging erasure-coding techniques to protect data. It also intelligently manages data placement across different media types, including HDDs and flash storage, to optimize I/O utilization for various tenants and workloads. This foundational layer is crucial for supporting the massive scale of Meta's operations, which includes hundreds of exabyte-scale storage clusters serving all of Meta's external and internal products, such as Facebook, Instagram, Reality Labs, Meta AI, and Ads. Building upon the Tectonic layer, Meta's BLOB-storage layers provide a globally accessible and infinitely scalable object storage fabric. This design allows for flexible policies that enable users to make trade-offs between data durability and availability, catering to diverse AI workload requirements. The architecture supports various storage APIs, including object storage, file systems, and block-device interfaces, all unified under this robust framework. The evolution of this BLOB-storage architecture was driven by the need for a step-function improvement in performance to effectively serve the demanding AI workloads. Meta's approach also considers the evolving constraints of datacenter operations, particularly the shift from space-constrained to power-constrained environments due to the high energy demands of GPUs. Every kilowatt of power spent on storage is power not available for GPUs, making power efficiency a new and critical constraint for AI workloads. The storage blueprint emphasizes cost-efficiency, moving beyond legacy HDD-centric optimizations to incorporate flash storage where necessary for IOPS-intensive AI workloads. This ensures that every kilowatt of power is utilized effectively, prioritizing GPU operations while still providing reliable and performant data access for the entire AI lifecycle, from training to inference. Furthermore, the architecture aims to maximize research velocity by minimizing the time researchers spend ingesting and moving data across regions, especially with geo-distributed GPUs and increasingly massive datasets.