Quite right. But the line between magic and telepathy is even thinner than that between science fiction and fantasy. The existence of telepathy has never been experimentally proven, nor has the existence of magic, and if telepathy or magic existed, people could use it to talk to each other mentally.
So in many cases the difference is just one of terminology. For example, there is the Darkover series of books by Marion Zimmer Bradley. It takes place on a lost colony planet where society and technology have been reduced to medieval levels, and some people have acquired abilities they call magic. As far as I remember, the reason for these abilities was their interbreeding with the native humanoids. If you take away the backstory about the lost colony and replace the native humanoids with some elves, you could easily turn the Darkover book series into a fantasy.