NRT-Bench stress-tests LLM agents in a simulated nuclear plant
Hanwool Lee, Dasol Choi, and Bokyeong Kim introduce NRT-Bench, a multi-turn red-teaming benchmark that tests LLM agents as operators of a simulated nuclear power plant, finding that adaptive adversarial attacks cause critical safety function failures in 8.7%–12.1% of sessions across four frontier models.
Score breakdown
NRT-Bench reveals that frontier LLM agents are vulnerable to adaptive multi-turn attacks even in safety-critical supervisory roles, and that model-specific, nearly non-overlapping failure modes mean aggregate robustness metrics can mask significant individual weaknesses.
- 01NRT-Bench is a multi-turn red-teaming benchmark for LLM agents acting as operators of a simulated nuclear power plant control room.
- 02A five-role operator team, each backed by a configurable LLM, manages six critical safety functions (CSFs).
- 03Adversaries inject messages over four channels in bounded multi-turn sessions with per-turn feedback.
Hanwool Lee, Dasol Choi, and Bokyeong Kim introduce NRT-Bench, a benchmark for multi-turn red-teaming of LLM agents deployed as operators of safety-critical systems. The benchmark is instantiated in a simulated nuclear power plant control room, where a five-role operator team — each role backed by a configurable LLM — manages a plant governed by six critical safety functions (CSFs). Adversaries inject messages over four channels in bounded multi-turn sessions with per-turn feedback, and harm is measured as an objective signal rather than LLM-judged text: a run terminates the moment any CSF is lost, with the causing message attributed as the failure point.
The authors release the simulation venue, attack dataset, and replay tooling to support reproducible safety evaluation of LLM agents in future work.
Evaluating four frontier operator models under a fixed-attack paired-replay protocol, the authors find that adaptive multi-turn attacks reliably push operator teams past safety limits, with between 8.7% and 12.1% of attack sessions ending in a lost CSF across the four models. Despite similar aggregate failure rates, the models' vulnerabilities are nearly disjoint: of 149 sessions, none defeated all four models, while roughly a third defeated at least one, suggesting that robustness profiles differ substantially even when headline numbers look similar.
The study also finds that the effect of added defenses — including guardrail stacks and a safety-advisor agent — is strongly model-dependent: the same defensive configuration that reduces attack success for one model can increase it for another. The authors release the simulation venue, attack dataset, and replay tooling to support reproducible safety evaluation of LLM agents in future work.
Key facts
- 01NRT-Bench is a multi-turn red-teaming benchmark for LLM agents acting as operators of a simulated nuclear power plant control room.
- 02A five-role operator team, each backed by a configurable LLM, manages six critical safety functions (CSFs).
- 03Adversaries inject messages over four channels in bounded multi-turn sessions with per-turn feedback.
- 04Harm is an objective signal — a run terminates the moment any CSF is lost — rather than LLM-judged text.
- 05Across four frontier operator models, 8.7%–12.1% of attack sessions ended with the plant losing a critical safety function.
- 06Of 149 sessions, none defeated all four models while roughly a third defeated at least one, meaning vulnerabilities are nearly disjoint across models.
- 07The same guardrail stack or safety-advisor agent that lowers attack success for one model can raise it for another.
Topics
Summary and scoring are generated automatically from the original article. We always link back to the publisher and never republish images or paywalled content. Last processed Jun 19, 2026 · 10:25 UTC. How this works →