I don't have much use for or interest in horror fiction as a genre. What I do value is the sense of dread and wonder that comes across in some novellas that could be called horror stories. Try Algernon Blackwood's "The Wendigo" or "The Willows," or Henry James's "The Turn of the Screw." For a science fiction variant, see Algis Budrys's "Rogue Moon" in The Science Fiction Hall of Fame, Vol. 2B. In short fiction, try A. M. Burrage's "One Who Saw" or some of M. R. James antiquarian tales. A science fictional variant in the short story form is Lovecraft's "Colour Out of Space." Why not start with the best -- stories such as these?
While I agree with you that a grounding in the work of the past is preferable, for many readers the way to the past is through the present: Many devotees of Stephen King were turned on to the works of past writers because of his
Danse Macabre.
All of the writers you mention, with the possible exception of the Budrys, are "horror". They may not fall into the publisher category, but literally for generations they've been the foundation of the ghost-/horror-story/weird tale and informed the work of many later writers. For instance, I think you can trace Machen in Elizabeth Hand's "Near Zennor" in the anthology Blacknorth and I discussed, as well as in Adam Nevill's
The Ritual; Nevill mentions both Machen and Blackwood in his acknowledgements along with James Dickey and Cormac McCarthy.
Post-
Rosemary's Baby, post-
The Exorcist, post-
The Other, post-
Carrie the publisher category of horror developed into a somewhat different thing, attracting a variety of writers in essence jumping in your face and shouting "BOO!" Some were effective (Poppy Z. Brite; Michael Blumlein; Kathe Koja, and probably several I'm not thinking of), often enough by not shouting, even though many writers were rather less successful.
Granted that this kind of story is rarely written the way Machen, Blackwood, the James boys, or HPL wrote it, that's probably not altogether bad. Ramsey Campbell wasn't really good until he broke away from a strict adherence to Lovecraftian style and approach; Thomas Ligotti is powerful because he merged interests beyond the Lovecraftian into his work; and I'd argue what I've read of Reggie Oliver is interesting because he translates the James (M. R.) ghost story into more contemporary subject matter with a more contemporary style. In particular, I would match the best of the short stories in Koja's
Extremities against the best by any writer you mention, and especially her story "Angels in Love."
I don't believe the problem currently is that there is nothing equaling the efforts of the past; instead I see the problem as sorting out from the massive amount of work available what is worthwhile from what is less so.
Randy M.