WEKA and OCI Validate 10x Throughput Gains for Long-Context AI Inference

In a significant development for artificial intelligence infrastructure, WEKA, a company specializing in AI data and memory solutions, has partnered with Oracle Cloud Infrastructure (OCI) to validate remarkable performance improvements for long-context AI inference. The companies recently published production-scale benchmarks showcasing how their combined technologies can dramatically enhance the efficiency and economics of running demanding AI workloads. The benchmarks, conducted on a nine-node OCI bare-metal H100 cluster, revealed that WEKA's NeuralMesh platform, equipped with its Augmented Memory Grid, can serve ten times more concurrent users and deliver ten times higher token throughput compared to traditional DRAM-only configurations. Furthermore, these tests demonstrated a sevenfold increase in the number of tokens processed per GPU, all achieved without the need for additional underlying infrastructure. This capability is particularly vital for enterprise AI applications that require extensive context windows and high GPU utilization. According to Pablo Selem, senior director of software development at Oracle Cloud Infrastructure, these benchmarks underscore how the NeuralMesh platform on OCI effectively eliminates memory bottlenecks. This allows customers to support larger and more complex inference workloads without simply resorting to adding more GPUs, thereby optimizing resource utilization and cost. The validation process involved testing with 100,000-token context windows and thousands of concurrent users, simulating real-world production environments. The results indicate a substantial shift in the economics of AI inference, making it more feasible and cost-effective for organizations to deploy and scale advanced AI models. By expanding the active cache working set from 8.64 TiB of DRAM to 287 TiB of usable NVMe, NeuralMesh with Augmented Memory Grid scaled past 5,000 concurrent users, significantly outperforming the approximately 600 users supported by DRAM-only setups. This innovation helps prevent the performance degradation that typically occurs when cache saturates. This collaboration between WEKA and OCI provides a clear path for enterprises to achieve higher performance and lower costs for their AI inference operations, especially as AI models continue to grow in complexity and demand for long-context processing increases. The NeuralMesh platform with Augmented Memory Grid is currently generally available to WEKA customers and can be found on the Oracle Marketplace, with OCI serving as WEKA's exclusive cloud launch partner for this technology.