StaminaBench stress-tests coding agents over 100 consecutive turns
StaminaBench is a new benchmark from Vlad Sobal, Shuo Yang, and Yuting Zhang that measures how many consecutive change requests a coding agent can handle before failing, revealing that all tested models collapse within 5–6 turns without test feedback.
Score breakdown
The benchmark exposes that current coding agents collapse within 5–6 turns on sustained multi-turn tasks — a failure mode invisible to single-task fraction-of-tasks-solved metrics — and quantifies that test feedback and harness choice are the dominant levers for improvement.
- 01StaminaBench measures how many consecutive change requests (turns) a coding agent handles before failing, rather than fraction-of-tasks-solved.
- 02Agents implement a REST API server and apply up to 100 procedurally generated change requests, producing codebases of up to 6,000 lines.
- 03Tests are generated fully programmatically without LLM involvement, ensuring reproducibility and reliability.
StaminaBench, introduced by Vlad Sobal, Shuo Yang, and Yuting Zhang, reframes how coding agent capability is measured by focusing on "stamina" — the number of consecutive interaction turns an agent can sustain before failing. The benchmark tasks agents with implementing a REST API server and then modifying it across a tunable sequence of procedurally generated change requests, set to 100 turns in the paper's experiments, resulting in codebases of up to 6,000 lines. Change sequences are drawn from either a hardcoded or LLM-driven sampler, both constrained to a structured action space to guarantee validity. Tests are generated fully programmatically without LLM involvement, ensuring reproducibility and reliability. The agent and server run in an isolated environment and communicate with the benchmark through HTTP, making evaluation fully black-box and language-agnostic.
The paper evaluates six agent harnesses paired with seven open-source LLMs across 20 scenarios of 100 turns each.
The paper evaluates six agent harnesses paired with seven open-source LLMs across 20 scenarios of 100 turns each. Three key findings emerge: first, all tested models fail within 5–6 turns when operating without thorough testing, confirming that vibe-coding-style programming accumulates bugs rapidly. Second, feeding test failure feedback back to the agent and allowing retries improves the passed turn count by up to 12x. Third, harness selection is critical — stronger models show up to a 6x gap between their best and worst harness, while weaker models fail regardless of harness. The benchmark code and generated tasks are released at github.com/amazon-science/StaminaBench to support further research into multi-turn coding agent behavior.
Key facts
- 01StaminaBench measures how many consecutive change requests (turns) a coding agent handles before failing, rather than fraction-of-tasks-solved.
- 02Agents implement a REST API server and apply up to 100 procedurally generated change requests, producing codebases of up to 6,000 lines.
- 03Tests are generated fully programmatically without LLM involvement, ensuring reproducibility and reliability.
- 04The benchmark is fully black-box and language-agnostic, communicating with agents via HTTP.
- 05Six agent harnesses paired with seven open-source LLMs were evaluated across 20 scenarios of 100 turns each.
- 06All tested models failed within 5–6 turns without test feedback; passing test feedback back improved passed turn count by up to 12x.
- 07Stronger models showed up to a 6x performance gap between their best and worst harness; weaker models failed with any harness.
Topics
Summary and scoring are generated automatically from the original article. We always link back to the publisher and never republish images or paywalled content. Last processed Jun 19, 2026 · 10:25 UTC. How this works →