Banning abuse against a minority group is clearly the right thing to do.
Banning people who some deem to express hatred of minorities is more problematic.
Many public figures have been accused (some more credibly than others) to oppress and spread hatred of minority groups.
Unless you want to make an endless series of dubious judgement calls, it’s often best to ban the issue (i.e. political discussions) rather than opinions within the issue.