Meta Unveils Llama 4 MoE Models Bringing Native Multimodality and 10M Context to Open AI

Meta unveiled the initial wave of its next-generation foundation suite with the release of Llama 4 Scout and Llama 4 Maverick, alongside technical previews of its flagship model, Llama 4 Behemoth. Departing from earlier dense architectures, Llama 4 transitions to a sparse Mixture-of-Experts (MoE) design featuring native multimodality trained on over 30 trillion tokens across text, image, and video data using FP8 precision. Llama 4 Scout couples 17 billion active parameters across 16 experts (109 billion total parameters) with an unprecedented 10-million-token context window engineered to fit on a single NVIDIA H100 GPU. Meanwhile, Llama 4 Maverick scales to 128 experts (400 billion total parameters with 17 billion active) to deliver advanced reasoning and coding capabilities distilled from the massive Behemoth base. This release fundamentally alters the operational economics of self-hosted enterprise AI. Historically, deploying frontier-grade multimodal reasoning required relying on proprietary managed APIs or maintaining prohibitive multi-node GPU clusters for monolithic dense weights. By constraining active compute to 17 billion parameters during inference while scaling total capacity across experts, Meta enables infrastructure teams to execute visual reasoning, large-scale codebase synthesis, and deep document extraction with significantly reduced serving latency and hardware overhead. Furthermore, Scout's massive 10-million-token window allows practitioners to ingest massive code repositories and historical log archives without relying on lossy retrieval chunking strategies. Architecturally, Llama 4 reflects the broader industry migration toward sparse routing and native multimodal pre-training. Following the scaling milestones established by Llama 3 and Llama 3.1, pure dense scaling encountered increasing thermal, bandwidth, and cost bottlenecks at inference time. By baking early vision-language fusion directly into the pre-training regimen rather than bolting adapter layers onto frozen text backbones, Llama 4 establishes a unified foundation comparable to closed frontier models while sustaining open-weights flexibility. For platform engineers and MLOps teams, deploying Llama 4 necessitates updating inference runtimes to handle MoE routing and dynamic expert dispatch efficiently. While Scout’s memory footprint is intentionally targeted at single-GPU deployments, serving Maverick and managing multimillion-token KV caches still require optimized memory management and high-bandwidth interconnects. DevOps organizations should audit their inference servers for native FP8 support, test quantization-aware pipelines, and validate internal data governance frameworks to leverage these models securely within air-gapped or virtual private cloud environments.