Aren't the Targaryens remembered as almost more than mortal? Because of their dragons and unusual physical traits, they were judged by different standards than other people. The lords of Westeros put up with a lot from the Targaryens, I think in part, so that they could boast that their kings were almost divine... But when Aerys started a bloodbath among the nobility and wanted to murder young men who'd not even entered the political arena, Jon Arryn put his foot down.
The lords of Westeros were of predominately four distinct ethnic origins and/or religions. The descendents of the First Men hold the north. They are dark haired. They worship the Old Gods. The descendents of the Andals are tall and fair haired. They rule the east, west, Riverlands, and the Reach. They worship the Seven. The descendents of the Rhoynar live in Dorne. For the most part they are dark, but some Dornishmen show interbreeding with the Andals. The Dornish worship the river goddess, if I recall correctly. The Ironborn rule the Iron Islands. I don't think they're ethnically related to either the Andals nor the Rhoynar... they might be related to the First Men, but that's a guess. Their religion is that of the Drowned God.
None of these peoples are Valyrian. None of them possess silver hair nor violet eyes. We're not told of the religion of Valyria, but presumably the Targaryens adopted the worship of the Seven when they conquered Westeros... or at least the Targs put up with the Faith. In fact, they abolished the military arm of the Faith (which Cersei just allowed to reform). I think that the conspicuousness of the otherness of the Targaryens allowed them great leeway in rule and living.
To support this, soon after Targaryen rule ended the Ironborn (twice), the First Men, many of the Andals, and even some Dornish wondered why they should be under the rule of the Iron Throne when a Targaryen is no longer sitting on it.
Edit: I forgot to say, "Welcome, Cerenna!"