If you want to assign more or less the same degree of implausibility to everything that's implausible, that's entirely up to you...So is FTL, materials stronger than chemical bonds, uploading minds, etc, etc. SF isn't about science, it is about possibility.
...but in the absence of evidence (and/or without a full understanding** of how our universe works), we have no idea which of them is or isn't possible (in some way, at least, or in certain circumstances)... which is why, though I don't think time travel is at all possible (and so see it as fantasy), I didn't say that it was conclusively impossible because I, like everyone else, don't know that for sure.
** - To go off at a tangent, we have no idea how existence... er... exists. It doesn't really matter whether it's meant to be via physics or (more or less) divine intervention, because the question then becomes one or more*** out of:
- from where did whatever the laws of nature that allowed our universe (with its physics) to come into existence come?
- from where did the God or gods who created our universe come?
- from where did those creating the simulation in which we might be "living" come?
*** - For all we know, it isn't turtles all the way down, but layers of turtles, elephants, etc., all the way down....