A former trustee at a charity co-founded by Prince Harry has said he is surprised at claims of bullying, harassment and misogyny made by its chairwoman.