My understanding is that Aegon the Conqeror laid the foundations of the Red Keep, his personal castle, at King's Landing. His son, Maegor the Cruel, finished the project and immediately murdered all of the workmen to keep the secret passages secret. I assume that this same Maegor made an addition to the original plans... Maegor's Holdfast.
The Eddard and Arya chapters of AGOT that depict the arrival of the Starks and their first weeks there give a good impression, excuse me... they formed my impression of the Red Keep. The Tower of the Hand is within the Red Keep. Remember during the siege, Sansa is able to move around from the Great Hall back to her chambers in the Tower of the Hand even after Cersei ordered the Red Keep sealed.
As to Joffrey's sword, I would expect that many people would forget it, yet certain others would not lose track of it. Auranne Waters left in secrecy, he might have pocketed it. Varys, Baelish, the High Septon, and even Pycelle might have spies watching it. I've always wondered about a discussion at Cersei's Small Council where she said Littlefinger requested some of Robert's old tapestries. Baelish got them at the end of AFFC if I'm not mistaken. The tapestries could have been a cover up for spies to steal the sword from KL. And then in AFFC, when the Lords Declarant meet with Littlefinger Sansa realizes that Lyn Corbray is working for Petyr and that Lyn likes to be paid with boys and/or swords.