The shadow home secretary has doubled down on Kemi Badenoch's criticism of the Conservatives' record on immigration, arguing "mistakes were made".