Randy, there are (so far as I know) just a few authors who wrote mostly or entirely in the vein of the antiquarian ghost story, such as James himself, A. N. L. Munby, and so on. Then there'd be authors who wrote some antiquarian ghost stories but also plenty of stories that wouldn't fit the definition. Russell Kirk would be an example.
Or take Lovecraft. It's a while since I read "The Shunned House," but I think that one might qualify as an antiquarian ghost story in the Jamesian sense. None of the "Cthulhu mythos" stories would, even though they may be loaded with antiquarian detail (e.g. "The Shadow Over Innsmouth").
Hmm -- okay, I think we could say this:
To qualify as a Jamesian antiquarian ghost story, a story must
--have a pronounced element of engagement with the past, involving tangible vestiges thereof; these might include manuscripts, artifacts (e.g. a whistle);
--suggest that some degree of scholarly learning is necessary for such resolution of the story as is to occur;
--observe a degree of reticence -- a Jamesian antiquarian ghost story cannot wallow in gore, and will never go in for explicit sexual content;
--include a haunter. However, this haunter need not be the shade of a human being, and indeed typically will not be such.
This last point might be worth discussing. Specifically, though the haunter must not be a living human being, must it be a supernatural being?
I'm thinking of the haunters in the Quatermass story about eerie phenomena, in which it transpires at last that the creatures responsible are Martians. Does the "science fiction" resolution mean that this story can't qualify?
I wouldn't be too quick to admit it to the canon of Jamesian antiquarian ghost stories. No, the more I think of it, the more I would disqualify it. The story eventually settles on a naturalistic explaining-away of the supernatural. It proposes that the devil and demons -- or, at least, the popular notion of horned beings as devils, which admittedly isn't necessarily the same thing -- are due to the Martians. I forget the details, whether there's an element of "race memory," & "telepathy," etc.
I would answer my own question above by saying that, yes, the haunter must be a supernatural being of some sort. It might be something that can be exorcised; it might be an "animated" dead thing that can be settled into death; but it can't simply be a thing that can be shot dead (unless maybe with a silver bullet?).
But now I'm wondering if the 20th-century "Martian" phenomena in the Quatermass story aren't represented as being psychic residua. Can we squeak them back in as "quasi" (whatever that means) supernatural phenomena? Obviously I'd need to refresh my memory of the Quatermass story. I think, though, that in the end I'm going to say it doesn't qualify as a Jamesian antiquarian ghost story (JAGS), but rather may be described as a science fiction story that cleverly uses elements akin to those of the JAGS.
So perhaps the criteria above will work for identifying a given story as JAGS or not, provided "haunter" is taken as something that retains its uncanny quality and is, perhaps, in fact, not susceptible of being quite pinned down according to the categories of daily rational inquiry.