The poll is the latest sign of mounting frustration with the billionaire entrepreneur as he's become a rising politic figure.