AWS Deception Benchmark Exposes High False Positive Rates in AI Security Models

AWS has released the Deception Benchmark alongside a research whitepaper to evaluate how effectively artificial intelligence models differentiate genuine security vulnerabilities from benign code that superficially resembles insecure patterns. Spanning 14,822 code samples across 16 programming languages and more than 70 Common Weakness Enumeration (CWE) categories, the benchmark assessed 12 frontier models from five providers. The findings revealed that under direct prompting, while models identified up to 95 percent of actual vulnerabilities, they also incorrectly flagged between 41 and 99 percent of safe code, resulting in precision rates clustering between 52 and 71 percent. When prompted for proof-of-exploit justifications, false positives fell significantly, but false negatives increased by 7 to 44 percent, and models routinely missed surrounding context such as Kubernetes network policies. This release highlights the central operational friction point in modern DevSecOps pipelines: alert fatigue and the erosion of engineering trust. Automated scanners that generate frequent false alarms waste on-call engineering cycles, delay software delivery, and lead developers to disregard legitimate security warnings. While offensive security demonstrations highlight automated exploit discovery in isolated functions, defensive reviews require contextual reasoning across enterprise codebases and deployment environments. For application security teams and cloud architects, AWS's empirical data demonstrates that general-purpose foundation models cannot yet reliably automate code vulnerability triage or pull-request approvals without substantial human oversight. The benchmark arrives amid an industry-wide push toward agentic AI tooling and automated assistants across the continuous delivery pipeline. As organizations increasingly integrate AI into code review and static analysis workflows, the common assumption has been that larger context windows and stronger reasoning models would naturally minimize spurious alerts. However, AWS's findings point to an intrinsic limitation: models frequently trigger on syntactic vulnerability signatures without validating whether application frameworks, runtime mitigations, or infrastructure configurations render the code safe. In practice, engineering leaders evaluating AI-driven security tools must demand transparent false positive and false negative performance data rather than relying solely on vulnerability detection rates. Security teams should avoid deploying direct LLM-driven gates that autonomously block developer pull requests or raise high-severity operational tickets. Instead, practitioners should integrate AI as an assistive layer with human-in-the-loop review on high-risk paths and supply full environmental context—including framework configurations and network security policies—when testing models against benchmarks like Deception Benchmark.