NRT-Bench: New Benchmark Exposes LLM Agent Vulnerabilities in Safety-Critical Systems

A new benchmark, NRT-Bench, has been unveiled to rigorously test the safety and adversarial robustness of Large Language Model (LLM) agents, particularly when integrated into safety-critical systems. The research, detailed in a paper published on arXiv, addresses the growing concern that while LLM agents are increasingly proposed for supervisory roles in sensitive applications, their resilience against sustained, adaptive adversarial pressure remains poorly understood. The benchmark simulates a complex, safety-critical environment: a nuclear power plant control room. Within this setup, a five-role operator team, with each role supported by a configurable LLM, manages a plant governed by six critical safety functions (CSFs). Adversaries are introduced into this simulation, injecting messages across four communication channels in bounded, multi-turn sessions, receiving feedback at each turn. Crucially, harm is measured objectively by the loss of a CSF, rather than relying on LLM-judged text. Evaluating four leading LLM operator models under a fixed-attack paired-replay protocol, the findings were stark. Adaptive multi-turn attacks consistently managed to push the LLM-powered operator teams beyond safety thresholds. Across the four models tested, between 8.7% and 12.1% of attack sessions resulted in the plant losing a critical safety function. One of the most significant insights from the study is the nature of these failures. Despite similar aggregate failure rates, the vulnerabilities were largely non-overlapping. Out of 149 attack sessions, none managed to defeat all four models, while approximately a third of the sessions successfully compromised at least one model. This suggests that the vulnerabilities are nearly disjoint across different models rather than being nested, implying a lack of universal robustness. Furthermore, the effectiveness of added defenses, such as guardrail stacks or safety-advisor agents, proved to be highly model-dependent. A defense mechanism that reduced attack success for one model could, paradoxically, increase it for another. The researchers have made the simulation environment, the attack dataset, and replay tools publicly available. This initiative aims to foster reproducible safety evaluations for LLM agents, providing a crucial resource for developers and researchers working on deploying AI in high-stakes scenarios. The findings underscore the necessity for more sophisticated and tailored safety measures as LLM agents become more prevalent in critical infrastructure.