Listed model price vs. actual run cost can diverge by 28x, study finds
A multi-institution study found that in more than one in five head-to-head model comparisons, the cheaper-listed model cost more to actually run — with the worst case reaching a 28x gap.
Score breakdown
The study shows that per-token list price is an unreliable proxy for actual LLM operating cost, with thinking-token variance and run-to-run randomness making true COGS unpredictable — a direct risk for any product built on flat-fee pricing over variable model usage.
- 01Study from Microsoft Research, Stanford, Berkeley, and CMU tested 8 frontier reasoning models across 9 task domains.
- 02In more than 1 in 5 head-to-head matchups, the cheaper-listed model cost more to actually run.
- 03Worst-case cost divergence between listed price and actual cost reached 28x.
A study from Microsoft Research, Stanford, Berkeley, and CMU ran 8 frontier reasoning models across 9 task domains, measuring not just listed per-token prices but the actual token consumption required to complete real work. The results challenge the common practice of selecting models based on pricing pages: in more than one in five head-to-head comparisons, the model with the lower listed price was more expensive to run. The worst-case divergence reached 28x. The most striking example cited is Gemini 3 Flash, which is listed 78% cheaper than GPT-5.2 but cost 22% more in practice across all tested tasks.
Thinking tokens made up over 80% of total output cost, and on the same query, one model used 900% more thinking tokens than another.
The core driver is thinking token consumption. Thinking tokens made up over 80% of total output cost, and on the same query, one model used 900% more thinking tokens than another. Compounding the problem, costs are not even stable: the same query sent to the same model produced bills that swung up to 9.7x between runs, meaning consumption is variable, model-specific, and partly random.
The post draws a direct implication for builders: COGS is not the sticker price, and any product charging a flat fee on top of variable LLM usage risks its heaviest users going underwater on margin. The distinction the post draws is sharp — the list price is a marketing number, while the actual bill is a behavior number — and the recommendation is to measure both before committing to a model.
Key facts
- 01Study from Microsoft Research, Stanford, Berkeley, and CMU tested 8 frontier reasoning models across 9 task domains.
- 02In more than 1 in 5 head-to-head matchups, the cheaper-listed model cost more to actually run.
- 03Worst-case cost divergence between listed price and actual cost reached 28x.
- 04Gemini 3 Flash is listed 78% cheaper than GPT-5.2 but cost 22% more across all tasks.
- 05Thinking tokens accounted for over 80% of total output cost.
- 06On the same query, one model used 900% more thinking tokens than another.
- 07Same query, same model: the bill varied by up to 9.7x between runs.
Topics
Summary and scoring are generated automatically from the original article. We always link back to the publisher and never republish images or paywalled content. Last processed Jun 20, 2026 · 08:55 UTC. How this works →