AWS Releases Deception Benchmark to Expose and Quantify LLM False Positives in Code Security

AWS Security researchers released the Deception Benchmark, an open-source evaluation suite and dataset comprising 14,822 code samples across 16 programming languages and more than 70 Common Weakness Enumeration (CWE) categories. Unlike traditional benchmarks that prioritize offensive exploit generation or raw vulnerability recall, this suite directly tests whether large language models (LLMs) can distinguish actual vulnerabilities from code that appears risky but contains legitimate mitigations. Across evaluations of 12 frontier models from five providers, standard prompting caught up to 95% of real vulnerabilities but generated false positive rates between 41% and 99% on safe code, leading to aggregate precision hovering in the mid-50% range. For SecOps, DevOps, and cloud architects, this evaluation surfaces the hidden operational tax of AI-assisted security tooling. While vendor demos celebrate high recall in scanning infrastructure-as-code or application logic, deploying high-noise models into continuous integration pipelines or security operations centers triggers alert fatigue and erodes developer trust. When security models flag secure patterns because they recognize a naive vulnerability signature without validating surrounding mitigations, developers spend valuable engineering cycles disputing non-actionable tickets. This benchmark aligns with a critical shift in cloud security engineering: transitioning from generative AI novelty to rigorous verification and grounded evaluation. As cloud platforms introduce agentic reasoning across IAM, static analysis, and runtime threat detection, single-pass pattern matching proves insufficient for complex systems. AWS's findings reveal that models struggle particularly with environment-gated controls—such as flagging vulnerable network code while completely ignoring neighboring Kubernetes Network Policies designed to block the vector. Prompting models for explicit proof-of-exploit reduced false positives by 17–74 percentage points, but at the cost of missing up to 44% of real vulnerabilities, underscoring fundamental architectural trade-offs in current LLM reasoning. In practice, security leaders and platform engineers should avoid implementing direct, unverified LLM gating in production CI/CD workflows. Organizations should require security vendors to benchmark false-discovery rates using objective suites like Deception Benchmark rather than reporting recall in isolation. Furthermore, teams must design multi-layered verification loops that enforce deterministic validation—such as policy-as-code engines and dynamic proofs—alongside human review for critical code paths, ensuring automated triage accelerates remediation without polluting operational backlogs.