I think people get "hero" and "protagonist" mixed up. The protagonist can be anybody, good, bad, or in between, so long as he is the focus of the story.
Stories are more likely to pull readers in if they have an emotional connection to the protagonist -- either to sympathize with, or to hate with a passion. But if a character is hateful, they need to have a sort of gruesome fascination about them. They can't be hateful in petty ways. It has to be something big, or they're just unpleasant to read about. Some redeeming quality, or if the character occasionally does something good or kind -- because he can, because it doesn't cost him anything or interfere with his bigger plans -- can add interest, because it will make him unpredictable, and leave readers in suspense about what he will do next.
There are so many stories where readers end up rooting for the villain instead of the hero (against the writers intentions) is because the hero is one-sided and predictable, and the villain is more human. But there is a danger the other way if we set out to make the bad guy the protagonist, because if he is too one-sided in his villainy, readers will become more interested in one of the good characters.
But for the original question: Where does the hero stop being the hero or even an antihero? That will be different for each reader. For me, it's not if he sometimes does unforgivable things, it whether or not he forgives himself. If he has no regrets, if he doesn't question what he's done, then I stop wanting to read about him. Horrible actions and a self-righteous attitude about them don't appeal. Besides, if a character blames himself, I, as the reader looking at it from all angles, am more likely to mentally argue in his defense, to think of all the reasons why it wasn't so bad. If he justifies himself, then I take the opposite side. His rationalizations sound weaker. I want a character to own what he has done and not pass over it glibly.
