RightersBlock, when springs says the show is lacking details, he means that you've never read nor heard the words "promise me, Ned", you've never heard of the Tower of Joy, you don't know what Jaime and Barristan thought of Rhaegar, you're not familiar with all the events of the Great Tourney of Harrenhal (Jaime's choice, Rhaegar's choice, and the story of the little crannogman), you've never been inside of Eddard's dreams, you don't know Eddard's regrets, you don't know the important things Dany saw in the Warlock's tower, you may not know the connection between Ser Jorah and the Old Bear, you may not have heard of the Doom of Valyria, you don't know the real story of Ramsey and Reek and Theon, you don't know that Ser Davos actually has seven sons, you don't know how funny Dolorous Edd can be, you don't know Ned Dayne, Rolph Spicer, Dontos Hollard, Black Walder, Maege Mormont, Balon Swann, Harwin, Jacelyn Bywater, May Stone, Bella, Merrett Frey, Small Paul, and many others.
You don't know the stories of the deceased... Lyanna Stark, Brandon Stark, Rickard Stark, Aerys II Targaryen, Rhaegar Targaryen, Joanna Lannister, Arthur Dayne, Ashara Dayne, Princess Elia, Princess Rhaenys, and Baby Aegon.
Also, the series is written in a peculiar method. Each chapter is told from one character's point of view. These POVs not only move the action, but allow the reader to understand the action through the eyes of a particular character. In A Game of Thrones, the Stark POVs dominate and we see the Lannisters as purely evil; the Greyjoys as renegades; the Baratheons as undisciplined (except for Stannis); the Martells as sneaky; the Tullys as honorable; and the Tyrells as opportunistic. But as the Lannisters, Greyjoys, and Martells gain more POVs, we begin to understand their motivations and we begin to see how they view the Starks... as poor, uneducated, bigoted, judgmental, inflexible, and over proud hillbillies.
Another thing... is that Season Three did not show all of A Storm of Swords (Book Three). If you started A Feast for Crows now, you would not know certain events in King's Landing, the Wall, and Mereen. You'll miss the Royal Wedding! And no one ever wants to miss one of those. You'll miss an epic duel of champions between a prince and a giant. You'll miss Mance Rayder's assault on The Wall... and a fight between a real giant and a blacksmith.
I know that you might think, "None of that seems important and besides I know the main story!" Well... I saw The Sixth Sense, and my thoughts went something like this, "I am following the plot... no problem... wait... WHAT?!?! Oh my goodness!!!! I never saw that coming." Now tell me truly, did you think Eddard was going to die? Or did you think he'd get out of it? Did you think Jaime would defenestrate Bran? Seriously, who starts a fantasy series by paralyzing one of the main candidates for adolescent hero? If you want the opportunity to see these things coming, then read the books.
You might also think, "Boaz, that's too much to remember. You're talking a cast of hundreds." Yes, I am. But if you don't get them fairly straight now, how are you going to learn them while simultaneously learning hundreds of new characters in AFFC and ADWD? The story just keeps getting bigger and bigger with at least ten new POVs in AFFC and ADWD.
My advice is to start now with AGOT and keep reading until Season Three starts, if you find no value to the books then move on to AFFC.
Edit: Plus, by starting at the beginning you'll learn Martin's style for this story. He hits the ground running and expects you to keep up. He does not really spoon feed his readers.