Browser-hardened frontier CUAs remain vulnerable in coding domain

Nicholas Saban's paper challenges the reliability of recently published red-teaming results for computer-using agents (CUAs), which have reported prompt-injection attack success rates (ASR) of 42–98%. The paper argues these headline numbers are inflated because they cluster on retired models and on the most-vulnerable model in each paper's panel. To test whether those techniques still work against current frontier CUAs using only hand-crafted templates, the paper releases CUA-HandCrafted — a public benchmark of 793 episodes spanning 24 multi-step web tasks, 56 attack templates, 8 attack families, and 4 system-prompt configurations.

A prompt ablation study shows this resistance is embedded in the model weights themselves rather than in system-prompt defenses.

Against Claude Sonnet 4.6 and GPT-5.4, the benchmark recorded 0/140 multi-step attack successes, with a Clopper-Pearson 95% upper bound of 2.60%. A prompt ablation study shows this resistance is embedded in the model weights themselves rather than in system-prompt defenses. However, this safety hardening does not generalize across domains: on a sister coding-agent benchmark called SkillBench, the same model weights are vulnerable to hand-crafted skill-injection attacks at success rates reaching up to 100%.

The paper concludes that the literature's high ASRs are largely attributable to RL-optimized injection text rather than the attack categories themselves, and that frontier safety hardening is domain-conditioned — specific to the browser surface, which has been heavily targeted and hardened. The paper also raises a reproducibility concern: when researchers report attack techniques without releasing the optimized injection strings, or extrapolate browser-domain safety findings to other CUA modalities, the published ASR numbers become unreproducible.

Key facts

1. 01The paper releases CUA-HandCrafted, a public benchmark of 793 episodes covering 24 multi-step web tasks, 56 attack templates, 8 attack families, and 4 system-prompt configurations.
2. 02Against Claude Sonnet 4.6 and GPT-5.4, the benchmark recorded 0/140 multi-step attack successes (Clopper-Pearson 95% upper bound: 2.60%).
3. 03A prompt ablation shows that browser-attack resistance lives in the model weights, not in system-prompt configurations.
4. 04On the coding-agent benchmark SkillBench, the same model weights fall to hand-crafted skill-injection attacks at up to 100% ASR.
5. 05Recent red-teaming literature reports prompt-injection ASRs of 42–98%, but these numbers cluster on retired models and the most-vulnerable model in each paper's panel.
6. 06The paper argues high published ASRs are largely attributable to RL-optimized injection text rather than the attack categories themselves.
7. 07Frontier safety hardening is described as domain-conditioned — specific to the heavily-targeted browser surface and not generalizing to other CUA modalities.

Topics