A general comment, not just on race, but on social identity more broadly...
People are people, first and foremost, and there's much more we all have in common than there is that sets us apart. I really HATE when writers ascribe "essential characteristics" or perspectives to individual characters based on those social categories, or treat characters from outside the dominant category like exotic bits of furniture. The first thing any author needs to do is portray ALL characters as individuals with true agency and subjectivity
But--like it or not--the circumstances of one's upbringing, the social categories adopted and imposed upon individuals, and the position of those social categories in a given society inform how individuals view the world and how others view that individual. Whether it's race, ethnicity, nationality, gender, sexuality, religion, class or migration status, these social circumstances do sometimes affect they way we interpret things, how we see things and how our actions are interpreted by others.
I think this is all very good, and I completely agree with it, but it can also be a source of 'unintentional' problems (i.e. those not caused by the unmediated prejudices of the writer, or their view of the reading public). Truly putting oneself in the shoes of one's characters (and not only the PoVs and important ones, but the minor ones as well) is difficult, particularly when the story requires information about a particular 'social group'** which is hard to obtain.
Yes, people are people, but as you say, all sorts of things influence how a person thinks and acts, and it's these differences that catch the readers' attention. (I think this is natural, and not specific to how we look at other humans.) Which means that those differences we do add call attention to themselves, and so need to be 'more right'*** than the more run-of the-mill stuff. And, of course, without proper research and attention to detail, any errors (from stuff that's simply wrong to the dependence on stereotypes) will leap out from the page to those in the know, and be added to the stereotypes for those who are as ignorant as the writer****.
All of which brings me to say that, in addition to prejudice, another reason that various 'social groups' are underrepresented could be the fear, in the writer's mind, that they'll get it wrong, and so they'd rather leave out characters from those 'social groups' to be on the safe side (and to save them much effort).
** - If that's the proper term.
*** - More right than one's perfect capturing of the mundane.
**** - I also wonder if poorly fleshed-out characters from various 'social groups' are sometimes seen as tokens, whether they are or not, because, in effect if not intention, they are. The solution here may be not to pick some random minor character to be of a 'different' 'social group', for whatever reason, but to have such persons as major characters (as in Stephen Palmer's
Muezzinland). If nothing else, that should justify putting the appropriate degree of research into these characters' motivations and character, which should at least help avoid the problems mentioned above.