Whether a Model Saves Tokens Is a Habit—Not How Hard the Task Looks

When one agent finishes a repo task for a fraction of another’s tokens, the easy story is “that task was easier for it.” A 2026 study of eight frontier models on OpenHands and SWE-bench Verified pushes a sharper claim: large efficiency gaps look like model habits—how they call tools and wander in files—more than how hard the issue is.

A scatterplot people can feel

Plotting average tokens against task accuracy, the study’s picture is blunt. OpenAI’s GPT-5 and GPT-5.2 land as high accuracy at relatively low token spend. Claude Sonnet 4 / 4.5 and Qwen3-Coder-480B sit in a higher-cost, mid-to-high accuracy band. Kimi-K2 shows the worst combination in the set: highest token use with the lowest accuracy. On the same 500 tasks, Kimi-K2 and Claude Sonnet 4.5 each averaged about 1.5 million more tokens than GPT-5.

Easy-for-all and hard-for-all still keep the ranking

To isolate difficulty, the researchers built two subsets: issues every model solved, and issues every model failed. Token-consumption rankings barely moved. That undercuts the story that “wasteful” models are only expensive because they face harder work. Their default loop—more redundant tool calls, more repeat file ops—shows up even when the task is easy, and they still struggle to stop early when the task is hopeless.

Failure has a surcharge—and the surcharge differs

• All models spent more tokens on failed tasks than on successes
• GPT-5 / GPT-5.2 added under about 500,000 tokens on failures; Kimi-K2’s failure premium reached about 2 million

Efficient models look better at noticing “this is not working” and stopping. Inefficient ones keep exploring until the budget is gone. For anyone picking models for agents, leaderboard accuracy alone is incomplete. You also want the habit profile: how much does it burn when it wins—and how badly does it thrash when it loses?