also, the engineer in me immediately thought that after a billion plus trips through the puzzle, surely at some point something would have gone wrong - the teleport room resetting itself incorrectly, the Doctor not making it back there, or something that would have screwed it all up. after all, no matter how small the chance, after that long,it's gonna happen. and, a billion skulls would have buried the castle... just saying
Give or take that the whole thing may not have been real, but a created reality; in which case, there weren't trillions** of Doctors (or trillions of skulls), just the one, one's whose few hours*** of memories were being wiped over and over again. (I don't think the use, by the writer, of the Retcon in the previous episode was random; not that I'm suggesting that Retcon itself was used, just the concept of wiping past "experiences".)
And also recall (pun intended) that the only reasons that the Doctor "knew" he had been there for a long time were: 1) the change in the positions of the stars****; 2) the change in the crystal. Both could have been manipulated. The only argument here is whether those playing with the Doctor wanted him to give up the Hybrid's location
by confessing or by any means available. And he did tell them anyway: he said it was him*****. (He may still have been lying, of course; but that links back to his behaviour in episode 2, with Davros: playing along with his enemy for his own purposes.)
Perhaps we should look back at this season's other episodes to see whether there are other hints as to what was really happening in this one.
** - He was there for, at most, a few hours, so could have gone round the loop almost seven hundred days per "year". (And that's if one assumes he was in a real place; but see ***.)
*** - Again, I don't think expanding a few moments in "reality" into minutes of monologue/dialogue on the mind-TARDIS was put there just to get Clara back in the episode: it was showing that time itself was being played with (but not in the usual timey-wimey way, but in the way we do when writing fiction).
**** - I would have thought that someone might have stumbled upon the tower during those billions of years, given that it remained in the same place. More likely that it didn't exist at all as a physical construction.
***** - I'm reminded of an episode ("The Interrogators") of the Avengers (Steed and Tara King) where agents are told (by enemy agents pretending to be part of the department) they're being tested to see whether they will give up any information. They don't, not until they're told the exercise is over, when they do so by accident when relaxing afterwards. (It has been on a couple of times over the last few months, on True Entertainment -- Channel 61 on Freeview.)