The government has been accused of "ignoring" the voices of people who lost family in the Grenfell Tower tragedy in its decision to demolish the building.