A housing minister has lost her building safety brief after Grenfell survivors complained about her attendance at a conference sponsored by a firm criticised by an inquiry into the deadly fire.