If It Feels Easy to a Human, That Still Won’t Predict the Agent’s Bill
Human time-to-fix ratings barely track agent token spend—and frontier models also systematically underestimate their own costs.
A natural hope is that experienced engineers can glance at a ticket and guess what an agent will spend—or that the agent can explore a bit and quote itself. Both hopes land weakly in a 2026 study of OpenHands agents on SWE-bench Verified.
Human difficulty is almost a different axis
On SWE-bench Verified, professional programmers rated issues by expected solve time: under 15 minutes, 15 minutes to 1 hour, or over 1 hour. Against actual agent token use, Kendall’s correlation was only about 0.32—very weak.
- About 6.7% of “under 15 minutes” tasks still spent more tokens than the average “over 1 hour” task
- About 11.1% of “over 1 hour” tasks spent less than the average “under 15 minutes” task
The mismatch is intuitive once you name it. A tiny syntax fix can force an agent to roam a large repo. A scary architecture question may be answered from pretrained pattern knowledge in a few turns. Human complexity and agent compute are not the same dial.
Letting the agent predict itself is only a coarse signal
The team redesigned the goal: explore the repo with tools, then estimate input, output, and total tokens instead of fixing the bug. Pearson correlations stayed weak to moderate. The best figure highlighted was Claude Sonnet 4.5’s output-token prediction at about 0.39; input predictions for most models clustered around 0.2–0.3.
Worse, models systematically under-predict—especially input tokens. Points sit far below the perfect-prediction diagonal even when real spend reaches millions of tokens. Removing in-context examples made underestimation worse, not better. The blind spot looks structural: models struggle to anticipate how long-context, multi-turn loops explode the input bill.
What still helps in products
Exact quotes before a run are a poor promise. Coarse signals still matter: budget caps, spend warnings, mid-run authorization, and pricing that admits agent volatility. Do not trust “this feels easy” as a cost forecast—and do not trust an agent’s self-estimate as a firm invoice either.