It's a good reasoning, but a couple of fact don't recommend to do that. First, government has to collect taxes and those poor people working now would have to pay them, decreasing consumption in ordinary goods, what destroys jobs as well. Second, if the poor people pay taxes and the rich don't do that, the inequality would increase rapidly.
Goverment has to find the balance in collection of taxes, because they are necessary in national defense, public security or unemployment insurance. It has to tax accordingly with the disposable income. And we have to consider that taxes create consumption and jobs as well, so tax the rich strongly than the poor is fair.
If the consumption of luxury goods decline and it results in hoard, I think that an income tax would be always better than a goods tax. But that hoarding could be reinvested in economic activity, creating more jobs as well.