I think its easier to turn our brains off for moving media which opens it up a lot more to diversity in quality and content; meanwhile for books we can't really turn our brain off as easily. Sure we can ignore inaccuracies and get swept along with the story; but its a much more thinking level of interaction.
It is a common misconception, and one that I myself was guilty of having several years ago.
How many novels have I browsed through, reading every single word on its pages and yet having no memory whatsoever of the previous 5 pages at any given time? It is just as easy to read a book without reading it than it is to watch a movie without watching it. Whenever I'm in bed and reading a book, I often find myself reaching the end of a page to realise that I have absolutely no idea about what that page was about (because I allow my mind to wander or am concerned about tomorrow's workload, or paying the bills, or whatever else...). The words enter my field of vision, I let their rhythm lull me to sleep, but I am not reading anything.
Reading a book "properly" and watching a movie "properly" demand just as much focus on the part of the reader/viewer. There is just as much subtext in a good movie as there is in a good book, and I often give out to people who claim to be cinephiles and consume 2 to 3 movies a day, but whenever they actually insert a Blu Ray into their player or - worse - turn on Netflix and play whatever happens to be the recommended movie of the day, they simply let the movie wash over them to play games on their iPads or use the movie as background noise. That is not what watching a movie is.
But in a society that has accepted movies as nothing more than "plots" read by pretty faces on a screen, I am not surprised that most people would have the notion that movies are to fast food what books are to 3-star restaurants.
In truth however, if you are not allowing the cinematography, sound design, production design and editing of a movie affect you (I am not saying you should be conscious of them and dissect the movie as you watch it, I am saying that you should be focused on the movie for these elements to have the effect the filmmakers designed them for), you are not truly experiencing the film, you're just following the plot (and in that case, why even bother to watch the film? Just read the script).