GLM 5.2 beats Claude Sonnet 4.6 on quality and cost in open-source coding agent benchmark

Nicolas Fortuin and Baptiste Fernandez evaluated four open-source coding models — GLM 5.2, MiniMax M3, Kimi K2.7-code, and Qwen3.7-Plus — against Claude Sonnet 4.6 using the publicly available `task-evals-for-skills` dataset. Each model solved nearly 1,000 real coding scenarios twice: a baseline run with the task alone, and a skill run that also provided an agent skill containing packaged conventions and instructions for the relevant tool. Scoring weighted instruction-following and task-completion at a four-to-three ratio in favor of instruction-following, on the premise that confidently completing the wrong thing is worse than stalling. Costs were computed from measured token counts at real list prices, with the four open models running on Fireworks at Standard rates and Sonnet 4.6 priced at Anthropic's list.

GLM 5.2 led the field with a 91.9 Overall score at $0.289 per task, narrowly beating Sonnet 4.6's 90.8 at $0.296 — making it both more accurate and cheaper.

GLM 5.2 led the field with a 91.9 Overall score at $0.289 per task, narrowly beating Sonnet 4.6's 90.8 at $0.296 — making it both more accurate and cheaper. MiniMax M3 scored 91.4 at just $0.207, yielding 442 points per dollar versus Sonnet's 307. Qwen3.7-Plus was the cheapest model by roughly an order of magnitude at $0.068 per task, but its 82.2 Overall score and instruction-following concerns limit its reliability. Kimi K2.7-code was the most expensive at $0.661 per task and ranked fourth in accuracy at 88.7. The skill consistently added about 20 Overall points across all models, with nearly all of that gain coming from instruction-following — the models could already complete most tasks, but lacked the conventions that skills provide.

Key facts

1. 01GLM 5.2 scored 91.9 Overall vs. Claude Sonnet 4.6's 90.8, at $0.289 vs. $0.296 per task.
2. 02MiniMax M3 scored 91.4 Overall at $0.207 per task, the best points-per-dollar among top performers (442 vs. Sonnet's 307).
3. 03Qwen3.7-Plus was the cheapest model at $0.068 per task but scored only 82.2 Overall.
4. 04Kimi K2.7-code was the most expensive at $0.661 per task and ranked fourth in accuracy at 88.7.
5. 05Adding an agent skill lifted Overall scores by ~20 points across all models, almost entirely through improved instruction-following.
6. 06Scoring weighted instruction-following over task-completion at a 4-to-3 ratio.
7. 07The evaluation used nearly 1,000 real coding scenarios from the publicly available `task-evals-for-skills` dataset.

Topics