I wonder if when we got into the middle parts of the 20th century that authors began to think they had to separate into different camps for some reason or the other so that now you wouldn't see "serious" authors write science fiction or fantasy.
No, and not quite.
This is a point which has been discussed a great deal over the years, and what it boils down to is: There has always been something of a dividing line between the literary classics and popular literature, though things would sometimes cross from one to the other (usually popular would become classic, not the other way around). This began to increase with a greater literacy rate; the more readers you have, the more people will read, and the majority (not being particularly well-educated, rather insular in their experiences with life, etc.) will read popular trash -- by which I do not mean simply popular literature, but things such as the "penny dreadfuls" and dime novels; stuff which was frankly often atrociously written, but very sensationalistic and therefore, like the tabloids of today, sold like hotcakes (as the saying goes). Opposed to that, you had writers who were actually aiming for self-expression, to address various issues they felt needed attention, or who simply wished to write as beautifully and as skillfully as they could.
For a very long time, this last class would also write things we would consider fantasy or horror or whatnot (Poe, Walpole, Dickens, Shelley, Byron, Keats, Brown, Hawthorne, etc., etc., etc.); but the divisions began to widen, as more literary critics began to make a distinction between "serious" literature and literature which relied on such fantastic tropes. This is largely the product of America (and, to a lesser degree, England), where such fabulist writing has always tended to be rather suspect; part of our Puritan heritage, many believe (and I tend to be one of them); the same sort of thinking which banned any sort of theatrical productions at the drop of a hat, because it was "frivolous" at best and "against the interests of the community" at worst.
Then came the pulp explosion, and increased marketing according to readers' tastes, and the result that almost universal literacy (albeit, in most cases, barely functional literacy) also meant that you had a tremendous audience, and not nearly enough quality writers to meet that demand... and so a correspondingly large number of mediocre writers or talentless hacks emerged to make a living... and the ghetto was well and truly born. Unfortunately, there are still a huge number of these sorts of writers writing in the genres and, as fans of these genres tend to be a bit more outspoken (and often less literate, in part because a lot of the noisier ones are, not surprisingly, in their teens), this gets more attention than the mediocre-to-bad writers of "mainstream" fiction. And, as those with less-developed tastes will often back anything they like, regardless of its actual artistic/aesthetic/literary merit... the genres are still looked on with a fair degree of disparagement... and, unfortunately, I'm afraid that is often well-deserved. One does not build a case for any type of work being as worthy as the genuinely lasting classics by being obstreperous, obnoxious, and undiscriminating in one's taste. One has to work at developing an understanding of the genuine literary canon and what separates these works from those which, whether they are excellent, good, mediocre, bad, or gawdawful, simply don't quite climb
that high on the mountain.
So, really, it was much less the writers (though there have been a fair number of those who felt they were "too good for" this or that genre, despite the fact that Shakespeare, Marlowe, Poe, Hawthorne, et al., often wrote in just these genres) than the marketers, publishers, critics, and reading public... and, of course, the educational community... which has rigidified that distinction.
But... that distinction still has a fair amount of truth to it. And here we're getting back to the main point of this thread: Speaking for myself, while I grew up reading an enormous amount more of generic literature (especially sf/f and horror, but also mysteries, adventure stories, etc.) than literary classics, I've always read the latter as well... and of late years, these have tended to predominate. And I find that, yes, they do have more to offer than the vast majority of writing in the genres. They are, by and large, better written, more polished, more refined in both language and thought, and usually deeper, with more layers to them, than generic literature. This is not to say that the genres don't offer such now and again -- they do -- nor that these exceptions don't deserve to eventually make it on that mountain, and perhaps even up with the best of them; but, being popular literature which is aimed at a popular audience and quite often written with considerably more haste and less deliberation, genre literature far too often tends to fall into sloppy writing, loose thinking, or just plain shallow effect and sensationalism rather than actually saying anything about what it means to be human on more than a very superficial level. These are the things the "literary classics" tend to offer more consistently than do even the classics of the genres, generally speaking.
However... as readers simply interested in entertainment, with or without the other aspects mentioned above, will always outnumber those who do look for these levels, generic literature is always going to have a wider appeal in most cases than will the classics. This is not an
insult to those who do read simply for entertainment; it is a
distinction between the purposes of reading, and should be taken as such. For me, personally, I can still enjoy a great deal of genre literature, but I no longer find it as satisfying on deeper levels as I do the genuine literary classics. However... the waters get muddy here, as a great number of the literary classics (including
Njal's Saga -- good choice, Extollager) also include elements of the fantastic, as it is only recently that these have been excluded from the literary mainstream; and even at that, they have never been successfully excluded for any length of time or in the great majority of works. (
Ulysses being an excellent example of an exception; as is
The Waste Land.....)