I read somewhere that it is a miscommunication due to different body language between species.
Someone who doesn't like cats will sit very still and hope the cat ignores them.
Someone who likes cats, but doesn't necessarily understand them (or at least nervous cats) will sit there going "Puss, Puss" and waving hands around. Confident cats are OK with that, nervous ones are not.
So in a room with a bunch of strangers, many cats (but not all) will prefer the sitting quietly non-scary person and avoid the noisy "puss, puss" person. And so the cat hater gets the cat.
I understood it was the eyes. Humans tend to narrow their eyes when looking at someone or something (eg a cat) they don't like. Cats on the other hand tend to narrow their eyes to show pleasure. So the cats see the narrowed human eyes of a cat hater as an invitation.