Well, to be succinct:
A) Interest.
B) Time.
(Yeah, I know. To quote Buffy: "Gee, could you vague that up for me?"
)
Nonfiction, it can be any number of factors, from simply a fascination with the subject (and its handling in that particular book or set of books) to a genuine admiration for and connection to the writer's style and approach.
For fiction, I would agree that there has to be some sort of connection to the characters or situation -- you don't necessarily have to like them, but you must be able to identify with them in some fashion; there has to be an honest expression of a human experience there, at least emotionally. No matter how entertaining a piece is, if it doesn't have that level of honesty in its handling, then it isn't likely to get people to reread it (generally speaking).
Note: I am not talking about an honest presentation of what it is like to actually, say, explore other worlds or have an encounter with a supernatural being, etc., but that the emotional responses -- not only of the characters, but those implied by the very word choice of the writing -- must convey a conviction and verisimilitude, an honesty about the sorts of emotions a reasonably intelligent adult human being (or child, should that be the case) would likely have feel in response to such a situation. In other words, an understanding of and empathy for people allowing the writer to present genuine, rather than false, stereotyped emotional responses to any given situation.
Aside from that... I have to feel that the book enriched me in some way; it caused me to reexamine my ideas on an aspect of life, or taught me something about the human condition, or deeply touched me in some manner. It goes beyond just entertainment and becomes art on some level. Again, this requires an emotional honesty and integrity on the part of the writer which is not always notable in fiction -- genre fiction especially.
And, I suppose, it has to, in some way, convey something about that writer's worldview; all the more so if it is taken in conjunction with other writings by that author. It has to say something about what he or she has learned about the human experience and our place in the world or cosmos. This does not have to be overt, but it does have to be present to give the work substance. The writer has to give something of him- or herself that transcends simple storytelling. One can see this sort of thing even in fairy tales or satires and parodies, let alone anything of a more "mature" manner.....
These, at least, are some of the things which draw me back to a book over time....