AWS Introduces Multi-Tier Storage Architecture for LLM Key-Value Caching

AWS detailed a reference storage architecture designed to optimize Large Language Model (LLM) inference workloads by introducing a four-tier Key-Value (KV) cache hierarchy across AWS Trainium (Trn2/Trn3) and NVIDIA GPU instances (P5, P5en, P6). The strategy structures data movement across High Bandwidth Memory (HBM), host memory/local NVMe storage, shared parallel file systems (Amazon FSx for Lustre), and object storage (Amazon S3) using open POSIX-compatible inference frameworks such as vLLM, SGLang, and LMCache. This architectural guidance matters because the cost model for generative AI has decisively shifted from model training to production inference. As enterprises adopt autonomous agents and long-context models handling prompts with hundreds of thousands of tokens, repeating the prefill phase across requests creates prohibitive latency and GPU spend. High Bandwidth Memory is severely capacity-constrained, making it impractical to pin persistent conversation contexts directly on accelerators. Offloading KV cache to tiered storage trades minimal network latency for massive compute savings, directly impacting the operational unit economics of multi-tenant model serving. This development fits into the broader convergence of high-performance cloud storage and distributed AI systems. In classical enterprise IT, tiered caching optimized transactional databases and file repositories; in modern AI infrastructure, high-throughput parallel file systems and low-latency object stores are becoming the de facto working memory for disaggregated inference architectures. By decoupling prompt processing (prefill) from generation (decode), storage tiers enable independent cluster scaling and shared context reuse across geographically distributed inference fleets. In practice, infrastructure teams should evaluate their serving framework's offloading capabilities and design storage tiers around request reuse rates. For applications characterized by shared system prompts, multi-turn agent dialogues, or document RAG workflows, platform engineers should deploy high-throughput local scratch tiers backed by FSx for Lustre rather than horizontally scaling raw accelerator instances. The primary trade-off involves managing promotion latency over microsecond budgets, necessitating proper bandwidth provisioning on host interconnects to prevent GPU memory starvation during KV cache restoration.