I have a massive problem with the concept of time travel both in fiction and in actuality.
It's the practical aspects that people forget, but lets pretend it is possible.
Here I am sat at my PC and wooosh back I go lets say 5 seconds.
Unfortunately in that time the Earth has rotated (1000mph) moved around the sun (67000mph) and the sun has nipped round the galactic centre at an estimated 200 Km/sec. We can ignore the galactic expansion/position change due to the expanding ever changing Universe stuff, lets not over complicate things. So when I arrive back in the past I'll probably live for about 5 seconds while I enjoy breathing the vacuum of space. In my last moments I can take in the view of the tiny blue dot, the Earth, speeding on it's way, whilst laughing it's polar cap off at my plight.
The alternative is that I've somehow managed to reverse "everything" in time. By everything I mean the whole Universe. That's quite some energy requirement from a 13 Amp plug.
Or I've managed to combine my time travel with the corrections required to arrive at my past self's position safely. However, if that's the case I'll be arriving there now and there's going to be an awful lot of bone crunching and blood splatter.
Lest we forget, this is true for me and for everything else, even a little old electron/photon/neutrino and the rest. Send them back in time and you won't be able to observe them, cos they'll not be there.
No use saying you sent them back and now their back and look, it's in a different position, because that position is way out of the observers field of view.
In this sense, Doc Who gets it right with the old TARDIS idea. And a Black hole for a power source at least tips it's hat to the energy required. However, one black hole a Universe shift does not make. All those other black holes are surely going to object to being bossed around by a little blue box.
So Time travel - ruins any plot for me.