FrontierCode benchmark tests if AI code is actually mergeable
Cognition introduced FrontierCode, a new coding benchmark built with open-source maintainers that evaluates whether AI-generated code is actually mergeable — not just unit-test passing — with the best model, Opus 4.8, scoring only ~13% on the hardest subset.
Score breakdown
FrontierCode directly addresses a documented flaw in existing coding benchmarks — that passing tests does not equal mergeable code — by introducing maintainability-focused evaluation criteria that reveal current frontier models are far from solving real-world code quality.
- 01Cognition introduced FrontierCode, a benchmark measuring whether AI code is actually mergeable, not just unit-test passing.
- 02Each FrontierCode task took 40+ hours to build with leading open-source maintainers.
- 03Evaluation dimensions include regression safety, cleanliness, scope, test correctness, and maintainability.
Cognition introduced FrontierCode, a new coding benchmark targeting a gap that existing evals like SWE-Bench have failed to capture: whether AI-generated code is actually mergeable, not merely capable of passing unit tests. Each task was constructed with leading open-source maintainers and required 40+ hours of work per problem. Evaluation criteria include regression safety, cleanliness, scope, test correctness, and maintainability. The benchmark was explicitly inspired by FrontierMath, which previously focused its hardest tier on problems that stumped frontier models. The headline result is that Opus 4.8, the best-performing model, scores only about 13% on the hardest subset — a stark contrast to the 50%+ scores models routinely achieve on SWE-Bench-style evals, suggesting coding is far less "solved" than popular benchmarks imply.
Several product changes reflect this trend, including observability dashboards for MCP connector developers and infrastructure for isolated, inspectable, long-running agent environments.
The article situates FrontierCode within a broader critique of existing coding benchmarks, noting that METR had previously found many SWE-bench-passing PRs would not be merged into main, and that the problem of false positive trajectories had been directly measured and addressed in the FrontierCode report. Beyond the benchmark itself, the article highlights a wider shift in the agentic coding space: practitioners are moving from one-shot prompts toward giving agents clear goals, verification criteria, and iteration structure. Several product changes reflect this trend, including observability dashboards for MCP connector developers and infrastructure for isolated, inspectable, long-running agent environments.
Key facts
- 01Cognition introduced FrontierCode, a benchmark measuring whether AI code is actually mergeable, not just unit-test passing.
- 02Each FrontierCode task took 40+ hours to build with leading open-source maintainers.
- 03Evaluation dimensions include regression safety, cleanliness, scope, test correctness, and maintainability.
- 04The best model, Opus 4.8, scores only about 13% on the hardest subset.
- 05FrontierCode was explicitly inspired by and named after FrontierMath.
- 06METR previously found that many SWE-bench-passing PRs would not be merged into main.
- 07A broader practitioner trend toward giving agents clear goals, verification criteria, and iteration structure was noted across the day's discussions.
Topics
Summary and scoring are generated automatically from the original article. We always link back to the publisher and never republish images or paywalled content. Last processed Jun 9, 2026 · 17:05 UTC. How this works →