It's not missing the point. It's just that the point is ridiculous and completely misses out any context or history of the use of the word.
It's because it has been used for decades/centuries(?) as a racist term to dehumanise black people and paint the whole race as thick, inferior, sub-human savages. And when they have actually been treated like that by governments, workplaces and pretty much everywhere up until not long ago, it's probably best not to say it.
Saying a white person, who looks like a monkey, looks like a monkey does not have these racist connotations so while it might hurt his feelings, it's not the same thing. It's hardly some injustice that is making all white peoples suffer, it's saying Gareth Bale looks like a monkey.