JAMER benchmark exposes AI's collapse on large game engine projects
Researchers introduce JamSet and JamBench, the first project-level game code dataset and benchmark built on the Godot engine, revealing that frontier AI models' runtime pass rates collapse from 80.4% on small projects to 5.7% on large ones.
Score breakdown
The benchmark reveals that frontier AI models — including those augmented with Code Agents — effectively fail at large-scale game project engineering, with runtime pass rates collapsing to 5.7%, exposing architectural design as an unsolved bottleneck that compilation-focused improvements cannot address.
- 01JamSet and JamBench are the first project-level game code dataset and benchmark built on a professional game engine (Godot).
- 02The dataset was sourced from Game Jam competitions, distilling 8,133 verified projects from over 240,000 repositories.
- 03300 manually verified projects form JamBench; the remaining verified projects constitute JamSet.
Jianwen Sun, Chuanhao Li, and Zizhen Li identify a significant gap in AI-driven game development research: while asset generation and web-based game coding have advanced, project-level code engineering on professional game engines has been largely unexplored due to the lack of large-scale datasets and deterministic evaluation methods. Their solution, JamSet and JamBench, is built on the Godot engine, leveraging its text-based format and headless execution mode to construct a deterministic verification pipeline that checks everything from file integrity to runtime behavior. The pipeline distills 8,133 verified projects from over 240,000 Game Jam repositories, with 300 manually verified projects reserved for JamBench.
A key finding is that Code Agents boost compilation rates but produce no improvement in runtime behavioral quality, suggesting the fundamental challenge is architectural design rather than syntactic correctness.
JamBench defines two task types — theme-driven generation and code completion — evaluated through a pipeline combining compilation pass rates, a Structural Completeness Score (SCS), and a Behavioral Alignment Score (BAS). Testing 9 frontier models reveals a dramatic capability cliff as project scale grows: runtime pass rates drop from 80.4% on small projects to 5.7% on large ones (Task2a). A key finding is that Code Agents boost compilation rates but produce no improvement in runtime behavioral quality, suggesting the fundamental challenge is architectural design rather than syntactic correctness. The authors also validate JamSet as effective training data, and all data and code are publicly available.
Key facts
- 01JamSet and JamBench are the first project-level game code dataset and benchmark built on a professional game engine (Godot).
- 02The dataset was sourced from Game Jam competitions, distilling 8,133 verified projects from over 240,000 repositories.
- 03300 manually verified projects form JamBench; the remaining verified projects constitute JamSet.
- 04JamBench evaluates models on theme-driven generation and code completion tasks using compilation pass rates, Structural Completeness Score (SCS), and Behavioral Alignment Score (BAS).
- 059 frontier models were evaluated, revealing a capability cliff: runtime pass rates drop from 80.4% on small projects to 5.7% on large ones (Task2a).
- 06Code Agents improve compilation rates but yield no gains in runtime behavioral quality, indicating the bottleneck is architectural design rather than syntactic correctness.
- 07All data and code are publicly available.
Topics
Summary and scoring are generated automatically from the original article. We always link back to the publisher and never republish images or paywalled content. Last processed Jun 19, 2026 · 10:25 UTC. How this works →